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Abstract 


The  Department  of  Defense  (DoD)  has  seen  unprecedented  growth  in 
spending  for  service  contracts  since  1990  (Ellman,  Livergood,  Morrow,  &  Sanders, 
2011)  during  the  same  period  in  which  there  has  been  a  general  reduction  in  the 
DoD  acquisition  workforce.  The  department  is  attempting  to  do  more  with  less,  year 
after  year.  The  level  of  scrutiny  focused  upon  DoD  service  contracts  by  the  upper 
echelons  of  the  DoD,  the  Government  Accountability  Office  (GAO),  and  Congress 
has  increased  as  the  spending  on  service  contracts  continues  to  increase  relative  to 
both  inflation  and  the  percentage  of  the  entire  DoD  contracting  budget  (Hart,  Stover, 
&  Wilhite,  2013). 

This  project  focuses  on  Army  service  contracts,  specifically  the  narrative 
section  of  contractor  performance  assessment  reports.  This  project  builds  upon  and 
advances  the  research  conducted  and  reported  in  the  Naval  Postgraduate  School 
MBA  research  project  Management  Levers  That  Drive  Services  Contracting  Success 
conducted  by  Hart  et  al.  (2013),  which  explored  the  relationship  between  CPARS 
objective  scores. 

This  report  focuses  on  the  quality  of  narratives  in  CPARS  and  their  value  to 
the  acquisition  process.  This  report  used  statistical  analysis  to  examine  715  Army 
service  contractor  performance  reports  in  CPARS  in  order  to  answer  the  following 
questions:  (1)  To  what  degree  are  government  contracting  professionals  submitting 
to  CPARS  contractor  performance  narratives  in  accordance  with  the  guidelines 
provided  in  the  CPARS  user’s  manual?  (2)  What  is  the  added  value  of  the 
contractor  performance  narratives  beyond  the  value  of  the  objective  scores  for 
performance?  (3)  What  is  the  statistical  relationship  between  the  sentiment 
contained  in  the  narratives  and  the  objective  scores  for  contractor  evaluations? 
Further,  contracting  professionals  were  interviewed  in  order  to  determine  answers  to 
the  following  two  additional  questions:  (4)  To  what  degree  do  the  interview  findings 
contradict,  support,  or  enhance  the  findings  for  the  three  previous  research 
questions?  (5)  What  conclusions  or  recommendations  can  we  draw  from  the 
answers  to  the  previous  research  questions? 

The  research  revealed  that  there  are  a  variety  of  opportunities  to  improve  the 
contracting  process  specifically  related  to  the  narrative  portion  of  past  performance 
assessment  reports. 

Keywords:  Contractor  Performance  Assessment  Reporting  System 
(CPARS),  Past  Performance  Information  (PPI),  Past  Performance  Information 
Retrieval  System  (PPIRS),  Service  Contracts,  Contract  Narratives,  Source 
Selection,  Contract  Administration 
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I.  INTRODUCTION 


A.  BACKGROUND 

The  purpose  of  this  chapter  is  to  introduce  the  system  designated  by  the 
Department  of  Defense  (DoD)  for  reporting  contractor  past  performance,  called  the 
Contractor  Performance  Assessment  Reporting  System  (CPARS).  The  DoD 
obligated  approximately  $360  billion  in  fiscal  year  2012  for  contracts  for  supplies  and 
services  (Government  Accountability  Office  [GAO],  2013).  In  order  to  support  best 
practices  for  government  acquisition,  the  Office  of  Federal  Procurement  Policy 
(OFPP)  directs  in  Federal  Acquisition  Regulation  (FAR)  Part  42  that  federal 
contracting  agencies  shall  use  CPARS  as  their  primary  tool  for  documenting 
contractor  past  performance. 

This  MBA  project  focuses  on  Army  services  contracts,  specifically  the 
narrative  section  of  contractor  performance  assessment  reports,  and  builds  upon 
and  advances  the  research  conducted  by  Hart,  Stover,  and  Wilhite  (2013)  which  was 
focused  on  the  CPARS  contractor  report  cards.  This  research  project  generated  a 
local  database  of  715  Army  service  contracts  which  were  used  to  correlate  the 
success  of  contracts  based  on  objective  scores  of  six  characteristics. 

This  is  the  eighth  research  project  in  a  series  of  MBA  projects  relating  to 
services  acquisitions.  Apte,  Dixon,  and  Rendon  are  collaborating  on  research  that 
focuses  on  the  value  of  CPARS  narratives,  the  correlation  of  narratives  to  objective 
scores,  and  the  narratives’  effect  upon  the  source  selection  process. 

Government  contracting  professionals  use  contractor  past  performance 
information  (PPI)  during  source  selection  and  contract  administration.  The  FAR 
requires  that  contracting  officers  consider  contractor  past  performance  as  one 
evaluation  factor  for  awarding  contracts.  Contractor  PPI  can  be  the  factor  that 
determines  whether  a  particular  contractor  will  be  awarded  government  business. 

B.  PROBLEM  STATEMENT 

CPARS  is  a  system  that  was  designed  to  be  used  by  contracting 
professionals  for  submitting  and  retrieving  contractor  past  performance  information. 
There  have  been  many  discrepancies  and  failures  specified  in  recent  GAO  reports 
with  the  utilization  of  CPARS.  Government  contracting  agencies  are  failing  to 
properly  document  contractor  PPI  within  the  schedule  requirements  mandated  by 
the  OFPP.  In  a  2009  report,  the  GAO  analyzed  data  from  2007  and  showed  that 
DoD  contracting  components  completed  required  contractor  past  performance  report 
cards  less  than  half  of  the  time  (GAO,  2009).  Subsequently,  the  DoD  increased  its 
focus  on  training  and  education  for  contracting  professionals,  which  led  to  an 
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increase  in  contractor  performance  assessments  being  completed  and  submitted  to 
the  Past  Performance  Information  Retrieval  System  (PPIRS).  In  the  last  published 
report  during  the  research  period  of  this  project  (GAO,  2013),  the  GAO  noted 
significant  gains  in  completion  rates.  Fifty-six  percent  of  required  reports  were 
completed  in  201 1 ,  while  74%  were  completed  in  2013.  However,  according  to  the 
same  report,  over  half  of  these  reports  were  submitted  late.  In  addition,  many 
CPARS  report  cards  contain  narratives  that  are  either  insufficiently  detailed  or 
conflict  with  their  associated  objective  scores.  Late  reports  lacking  sufficient 
accurate  information  provide  less-than-optimal  information  to  the  contracting 
professionals  who  rely  on  these  report  cards  for  source  selection  and  contract 
administration  purposes. 

C.  PURPOSE 

Government  contracting  professionals  submit  to  CPARS  information  on 
contractors  related  to  their  contract  performance.  This  information  is  in  the  form  of 
objective  scores  in  five  categories:  Quality,  Schedule,  Cost  Control,  Business 
Relations,  and  Management  of  Key  Personnel.  In  addition  to  these  five  categories, 
there  is  a  specific  contractor  evaluation  section  in  which  the  government  evaluator 
writes  a  descriptive  narrative  of  the  contractor’s  performance. 

The  purpose  of  this  research  is  to  attempt  to  determine  the  value  of  contractor 
performance  assessment  report  narratives  for  services  contracts  by  comparing  the 
relationships  between  narratives  and  objective  scores  in  order  to  recommend 
improvements  to  the  CPARS  contractor  performance  information  documentation 
process.  These  recommendations  regarding  the  process  aim  to  improve  the 
CPARS  report  card  product,  which  should  lead  to  greater  and  more  effective 
utilization  of  the  CPARS  system  for  source  selection  and  contract  administration 
purposes. 

D.  RESEARCH  QUESTIONS 

The  primary  question  addressed  by  this  research  project  is  as  follows:  Does 
the  CPARS  report  card  written  narrative  section  provide  value  to  the  contractor 
performance  evaluation  process?  This  research  focuses  on  five  research  questions 
that  explore  the  nature  of  CPARS  narratives: 

1 .  To  what  degree  are  government  contracting  professionals  submitting 
contractor  performance  narratives  to  CPARS  in  accordance  with  the 
guidelines  provided  in  the  CPARS  user’s  manual? 

2.  What  is  the  added  value  of  the  contractor  performance  narratives 
beyond  the  value  of  the  objective  scores  for  performance? 
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3.  What  is  the  statistical  relationship  between  the  sentiment  contained  in 
the  narratives  and  the  objective  scores  for  contractor  evaluations? 

4.  To  what  degree  do  the  interview  findings  contradict,  support,  or 
enhance  the  findings  for  the  three  previous  research  questions? 

5.  What  conclusions  or  recommendations  can  we  draw  from  the  answers 
to  the  previous  research  questions? 

E.  METHODOLOGY 

This  research  examines  the  value  of  CPARS  report  card  narratives  for  service 
contracts  as  they  relate  to  their  associated  objective  scores.  The  methodology 
includes  a  literature  review,  data  analysis  (including  both  sentiment  and  statistical 
analysis),  and  interviews  with  government  agency  contracting  professionals. 

The  literature  review  was  conducted  for  the  purpose  of  examining  and 
considering  current  knowledge  available  from  the  existing  body  of  literature  on  the 
subjects  of  federal  contracting  and  government  contractor  performance.  The  GAO 
and  OFPP  have  released  several  studies  and  memoranda  on  the  subject  of 
contractor  past  performance  as  it  relates  to  the  government  contracting  process. 

This  research  was  based  in  large  part  upon  these  documents  and  previous  research 
performed  by  Hart  et  al.  (2013)  that  explored  the  drivers  of  successful  service 
contracts.  Building  on  their  research,  this  project  conducted  a  review  of  literature 
about  the  service  contract  management  process  and  the  CPARS. 

The  first  phase  of  data  analysis  performed  was  a  sentiment  analysis  of  715 
Army  service  contract  CPARS  report  card  narratives.  The  CPAR  Ouality  Checklist 
was  used  as  a  basis  for  developing  the  criteria  for  the  categories  and  values  for  the 
analysis  (“CPAR  Ouality  Checklist,”  n.d.).  Narratives  were  assigned  several  scores 
relating  to  their  quality,  robustness,  and  compliance  with  directions  in  the  CPAR 
Ouality  Checklist,  and  to  their  value  and  content  compared  to  their  related  objective 
scores  from  the  CPARS  report  cards. 

The  second  phase  of  data  analysis  included  a  statistical  analysis  of  the 
relationship  between  the  scores  provided  by  the  sentiment  analysis  and  their 
associated  objective  scores.  This  analysis  looked  at  correlating  relationships 
between  the  various  sentiment  scores  and  objective  scores  for  the  same  CPARS 
report  to  find  meaningful  relationships  between  them  and  to  reveal  the  extent  of  the 
value  of  the  narratives. 

In  support  of  the  data  analysis,  interviews  with  contracting  professionals  from 
two  DoD  contracting  agencies  were  conducted.  These  interviews  consisted  of  a 
series  of  questions  asking  the  subjects  how  they  use  and  to  what  extent  they  value 
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CPARS  and  other  sources  of  contractor  past  performance  information  and  to  what 
extent  they  value  the  narratives  compared  to  the  performance  objective  scores. 

F.  BENEFITS  AND  LIMITATIONS 

The  purpose  of  this  research  is  to  attempt  to  determine  the  value  of  contractor 
performance  assessment  report  narratives  for  services  contracts  by  comparing  the 
relationships  between  narratives  and  objective  scores.  Potential  benefits  to  this 
research  include  a  better  understanding  of  the  value  of  the  CPARS  narratives  and 
the  extent  to  which  DoD  contracting  professionals  value  them  relative  to  other 
sources  of  contractor  past  performance  information.  This  study  provides  information 
based  on  data  indicating  how  closely  written  narratives  correlate  to  their  associated 
objective  scores.  It  also  shows  how  well  DoD  approving  officials  and  the  individuals 
contributing  to  the  narratives  comply  with  the  directions  for  completing  a  CPARS 
narrative  as  instructed  in  the  CPAR  Quality  Checklist  (“CPAR  Quality  Checklist,” 
n.d.). 

There  are  clearly  identifiable  limitations  to  the  research  being  conducted. 

First,  this  research  project  is  limited  to  the  715  Army  service  contracts  analyzed  by  a 
previous  Naval  Postgraduate  School  (NPS)  MBA  project  (Hart  et  al. ,  2013).  The 
selection  criteria  for  those  715  contracts  were  limited  to  Army  service  contracts, 
“contracts  only  submitted  by  Mission  Installation  Contracting  Commands  (MICCs), 
and  contracts  from  only  five  MICC  offices”  (Hart  et  al.,  2013,  p.  5). 

The  next  limitation  is  the  dollar  threshold  for  reporting  contractor  performance 
information  in  CPARS.  DoD  CPARS  policy  provides  different  dollar  thresholds  for 
mandatory  reporting  of  contractor  PPI.  Service  contracts  must  report  contractor  PPI 
starting  at  dollar  amounts  greater  than  $1  million  (Department  of  the  Navy  [DoN], 
1997).  This  means  contractor  PPI  for  service  contracts  may  not  be  reported  if  dollar 
amounts  are  below  $1  million.  This  limits  the  research  database  to  either  service 
contracts  above  $1  million  or  service  contracts  below  the  threshold  that  DoD 
contracting  professionals  chose  to  enter  into  CPARS. 

Another  limitation  of  this  research  is  the  fact  that  DoD  contracting 
professionals  might  not  enter  PPI  into  CPARS  even  though  it  is  required.  As  of  the 
completion  of  this  project,  the  most  recent  report  (GAO,  2013)  shows  this  lack  of 
performance  with  56%  and  74%  of  CPARS  reports  completed  in  201 1  and  2013, 
respectively.  These  limitations  are  discussed  further  in  later  chapters. 

G.  SCOPE  AND  ORGANIZATION  OF  RESEARCH 

This  research  report  consists  of  five  chapters  organized  in  the  following 
manner.  Chapter  I  contains  background  information,  a  problem  statement,  the 
purpose  of  the  research  project,  research  questions,  the  methodology  of  research, 
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the  benefits  and  limitations  of  the  project,  and  the  scope  and  organization  of  the 
research  project.  Chapter  II  is  a  literature  review  that  provides  an  overview  of  the 
DoD  acquisition  process  for  service  contracts,  CPARS,  the  increase  in  scrutiny  of 
federal  contracting  agencies,  and  previous  studies  that  led  to  this  research.  Chapter 
III  contains  the  methodology  for  how  the  research  was  performed,  organized,  and 
analyzed.  This  chapter  also  discusses  the  purpose  and  scope  of  interview  questions 
posed  to  DoD  contracting  professionals.  Chapter  IV  provides  an  analysis  of  the  data 
and  presents  the  answers  to  the  research  questions.  Chapter  V  summarizes  the 
research  project,  provides  conclusions,  and  addresses  potential  areas  for  additional 
research. 

H.  SUMMARY 

This  chapter  provided  the  background  information  leading  up  to  the  research 
project,  a  problem  statement,  the  purpose  of  the  research  project,  research 
questions,  the  research  methodology  used  to  analyze  the  data,  benefits  and 
limitations  of  the  research,  and  the  scope  and  organization  of  the  overall  project. 

The  next  chapter,  Chapter  II,  provides  the  literature  review  of  the  DoD  acquisition 
services  contracting  process,  the  CPARS,  and  previous  studies  conducted  by  NPS 
in  relation  to  service  contracts. 
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II.  LITERATURE  REVIEW 


A.  INTRODUCTION 

The  purpose  of  this  chapter  is  to  provide  an  introduction  to  the  DoD 
acquisition  process  for  services,  with  a  focus  on  Army  services  contracts.  This 
review  attempts  to  familiarize  the  reader  with  the  necessary  information  to 
understand  the  contracting  structure,  contracting  terms,  roles  and  responsibilities  of 
the  acquisition  team,  and  the  system  that  the  DoD  has  implemented  for  documenting 
and  retrieving  contractor  past  performance  information,  called  the  Contractor 
Performance  Assessment  Reporting  System  (CPARS).  This  chapter  provides  a 
discussion  about  the  purpose  of  CPARS  as  well  as  current  issues  associated  with 
CPARS.  The  chapter  includes  a  look  at  reports  submitted  by  a  number  of 
government  authorities  that  point  out  the  successes  and  failings  of  various 
government  contracting  agencies  in  complying  with  laws  and  directives  specifying 
how  and  to  what  extent  they  are  to  utilize  CPARS  for  documenting  contractor  past 
performance  information. 

B.  THE  DOD  ACQUISITION  PROCESS  FOR  SERVICE  CONTRACTS 

To  understand  the  DoD  acquisition  process,  the  reader  first  needs  to 
understand  what  the  term  acquisition  means.  The  Federal  Acquisition  Regulation 
(FAR)  defines  the  term  acquisition  as  follows: 

acquiring  by  contract  with  appropriated  funds  of  supplies  or  services 
(including  construction)  by  and  for  the  use  of  the  Federal  Government 
through  purchase  or  lease,  whether  the  supplies  or  services  are 
already  in  existence  or  must  be  created,  developed,  demonstrated, 
and  evaluated.  Acquisition  begins  at  the  point  when  agency  needs  are 
established  and  includes  the  description  of  requirements  to  satisfy 
agency  needs,  solicitation  and  selection  of  sources,  award  of 
contracts,  contract  financing,  contract  performance,  contract 
administration,  and  those  technical  and  management  functions  directly 
related  to  the  process  of  fulfilling  agency  needs  by  contract.  (FAR 
2.101(b)) 

The  basic  definition  is  that  the  government  has  a  need  and  fulfills  that  need 
by  purchasing  a  product  or  service  through  a  defined  process  using  a  set  of  rules. 

All  DoD  acquisitions — services  and  non-services  alike — begin  the  contracting 
process  with  the  same  fundamental  step:  identify  a  bona  fide  need  or  requirement. 
The  customer,  usually  an  organization  or  an  entity  within  an  organization,  identifies  a 
product  or  service  that  is  essential  for  that  organization  to  complete  its  mission  or 
function.  The  customer  must  be  specific  when  describing  the  agency  need  for  the 
contracting  officials  to  accurately  identify  the  requirement  and  begin  the  contracting 
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process.  According  to  FAR  Part  1 1 ,  Describing  Agency  Needs,  the  customer  must, 
to  the  maximum  extent  possible,  adhere  to  the  following: 

(i)  State  requirements  with  respect  to  an  acquisition  of  supplies  or 
services  in  terms  of  - 

(A)  Functions  to  be  performed; 

(B)  Performance  required;  or 

(C)  Essential  physical  characteristics.  (FAR  1 1 .002) 

An  example  of  a  bona  fide  need  is  as  follows.  The  Army  Mission  and 
Installation  Contracting  Command  (AMICC)  received  a  request  from  a  customer  to 
purchase  janitorial  services.  The  customer  has  funding  available  in  the  current  fiscal 
year  and  is  requesting  12  months  of  services  for  one  building  with  a  requested 
contract  start  date  two  months  from  the  submission  of  the  requirement.  The 
services  consist  of  the  following  tasks:  sweep  floors  daily,  mop  floors  with  soap  and 
water  one  time  per  week,  wax  floors  one  time  per  quarter,  remove  trash  from 
garbage  bins  every  Thursday,  clean  and  sanitize  bathrooms  daily,  and  other 
specified  tasks.  The  service  requirement  is  clearly  defined  with  funding  available  to 
obligate  expenses  and  provides  the  contracting  official  with  sufficient  information  to 
begin  the  acquisition  process,  and  so  a  bona  fide  need  is  apparent. 

The  following  request  for  maintenance  services  would  be  rejected  because  it 
does  not  meet  the  bona  fide  need  requirement.  The  AMICC  receives  a  request  for 
M1A1  Abrams  tank  maintenance  services  from  Fort  Drum’s  10th  Mountain  Division. 
This  request  would  be  denied  as  the  10th  Mountain  does  not  have  tanks  on  its  Table 
of  Organization  and  Equipment  (TO&E).  A  unit  cannot  request  maintenance 
services  for  equipment  it  does  not  have.  Alternatively,  a  request  by  the  U.S.  Army 
Armor  School  at  Fort  Knox  might  have  a  bona  fide  need  for  tank  maintenance 
services  because  they  train  Army  and  Marine  Corps  personnel  on  the  Abrams  tank. 

Once  a  bona  fide  need  has  been  identified,  the  service  acquisition  process 
can  begin.  The  service  acquisition  plan  is  a  comprehensive  strategy  that  identifies 
the  relationship  between  the  plan’s  phases  and  the  work  to  be  performed,  lists  the 
milestones  to  be  completed,  and  provides  the  overall  approach  that  management 
will  follow  to  mitigate  risk  while  meeting  the  service  requirements  (Neuman,  2013b). 

The  service  acquisition  process  can  be  identified  by  its  three  phases:  Plan, 
Develop,  and  Execute.  These  phases  break  down  into  seven  distinct  processes  that 
build  upon  the  previous  process.  As  Beers  (201 1)  wrote,  “The  Planning  phase,  as 
depicted  in  Steps  1 , 2,  and  3,  lays  the  foundation  for  the  services  acquisition”  (p.  56). 
According  to  Beers  (2011),  “During  the  development  phase — Steps  4  and  5 — a 
requirements  roadmap  process  is  used  to  define  performance  objectives  and 
standards,  allowable  variations,  and  method  of  performance  assessment.  In  the 
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execution  phase — Steps  6  and  7 — the  team  puts  all  the  customer’s  planning  and 
development  efforts  into  action”  (p.  57).  Figure  1  explains  the  seven  stages  of  the 
service  acquisition  process. 


Execute 


•  Sdect  Right  Contractor 

•  Award  Contract 

•  Roll  Out  Strategy 


Figure  1.  The  Services  Acquisition  Plan 

(Beers,  2011,  p.  56) 


Phase  1:  Plan 

The  first  phase  of  the  services  acquisition  process  is  the  planning  phase. 

This  phase  is  extremely  important  as  it  is  the  foundation  for  the  service  acquisition. 
The  planning  phase  contains  three  steps:  (1 )  form  the  team,  (2)  review  current 
strategy,  and  (3)  market  research. 

Step  1:  Form  the  Team 

During  Step  1  of  this  phase,  the  agency  head  is  responsible  for  designating 
who  will  be  a  member  of  the  acquisition  team,  providing  them  with  the  guidance  on 
how  to  proceed,  and  ensuring  that  the  plan  complies  with  FAR  requirements  (FAR 
7. 1 03(i)).  The  acquisition  team  consists  of  anyone  who  will  be  responsible  for 
significant  portions  of  the  service  acquisition  throughout  its  life  cycle  (FAR  7.102(b)). 
No  single  person  has  the  requisite  knowledge,  technical  proficiency,  time,  or 
resources  to  perform  all  the  functions  necessary  to  plan,  source,  award  and 
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administer  a  service  contract,  but  an  acquisition  team  made  up  of  experts  in 
functional  areas  can.  Acquisition  team  members  can  include,  but  are  not  limited  to, 
the  program  manager,  contracting  officer,  contracting  officer’s  representative  (COR), 
and  others  with  expert  functional  area  knowledge  like  finance,  legal,  engineering, 
small  business  administration,  quality  assurance,  logisticians,  and  the  customer/end 
user  (DoD,  2012). 

Program  Manager:  The  program  manager  (PM)  is  the  single  person  tasked 
with  the  responsibility  to  manage  the  acquisition  team  while  balancing  the  acquisition 
cost,  schedule,  and  performance  for  that  program  (Neuman,  2013a).  PMs  perform 
essential  functions  such  as  attending  program  reviews,  defending  the  program  or 
fighting  for  additional  funding,  fighting  requirements  creep,  and  coordinating  with 
senior  DoD  and  congressional  staffers  to  ensure  compliance  with  statutory 
requirements  and  congressional  intent.  Given  the  importance  of  the  PM  position,  it 
is  unfortunate  that  previous  research  by  Apte  and  Rendon  (2007)  found  that  “in 
many  service  acquisitions,  a  project  manager  or  project  team  is  not  assigned  and 
the  contracting  officer  assumes  the  responsibilities  of  the  project  manager  or  project 
team  leader”  (Hart  et  al. ,  2013,  p.  10). 

Contracting  Officer:  The  contracting  officer  (KO),  as  defined  by  Neuman 
(2013b),  is 

responsible  for  providing  contracting  support  required  for  meeting  the 
project’s  cost,  schedule,  and  performance  objectives  while  in 
compliance  with  the  statutory  requirements  and  agency  regulations 
ensuring  performance  of  all  necessary  actions  for  effective  contracting, 
ensuring  compliance  with  the  terms  of  the  contract,  and  safeguarding 
the  interests  of  the  United  States  Government  in  its  contractual 
relationships,  (p.  34) 

Basically,  the  KO  works  to  support  the  PM  to  meet  the  program’s  goals,  but 
the  KO  must  adhere  to  all  laws,  regulations,  and  agency  policies. 

Contracting  officers  have  specific  responsibilities  defined  by  regulations. 
According  to  FAR  1 .602-2,  a  contracting  officer  shall 

a)  Ensure  that  the  requirements  of  1 .602-1  (b)  have  been  met,  and 
that  sufficient  funds  are  available  for  obligation; 

b)  Ensure  that  contractors  receive  impartial,  fair,  and  equitable 
treatment; 

c)  Request  and  consider  the  advice  of  specialists  in  audit,  law, 
engineering,  information  security,  transportation,  and  other 
fields  as  appropriate; 

d)  Designate  and  authorize,  in  writing  and  in  accordance  with 
agency  procedures,  a  contracting  officer’s  representative  (COR) 
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on  all  contracts  and  orders  other  than  those  that  are  firm-fixed 
price,  and  for  firm-fixed-price  contracts  and  orders  as 
appropriate,  unless  the  contracting  officer  retains  and  executes 
the  COR  duties.  (FAR  1 .602-2) 

Army  contracting  officials  can  be  either  civilian  employees  authorized  to  serve 
in  1102  positions  (Office  of  Personnel  Management  [OPM],  1983)  or  military 
personnel  serving  in  the  following  positions:  Functional  Area  97A,  Contracting  and 
Industrial  Management  Officer  (Jones,  2007)  or  Military  Occupational  Specialty  51 C, 
Acquisition,  Logistics,  and  Technology  Contracting  Non-Commissioned  Officer  (U.S. 
Army,  2011). 

In  the  past  decade,  Army  acquisition  officials  have  increasingly  operated  in 
joint  environments.  This  has  required  Army  contracting  personnel  to  interact  with 
contracting  specialists  from  the  other  services  and  other  governmental  agencies. 
While  all  DoD  personnel  operate  under  the  same  FAR  and  Defense  Federal 
Acquisition  Regulation  Supplement  (DFARS),  each  DoD  organization  has  its  own 
supplement  to  the  FAR,  and  each  organization  outside  the  DoD  may  operate  under 
different  guidance  promulgated  by  its  parent  organization  (e.g.,  Army  FAR 
Supplement  [AFARS],  Air  Force  FAR  Supplement  [AFFARS],  Navy  Marine  Corps 
Acquisition  Regulation  Supplement  [NMCARS],  Department  of  State  Acquisition 
Regulation  [DOSAR],  and  Department  of  Veterans  Affairs  Acquisition  Regulation 
[VAAR];  FAR  1.301). 

Contracting  Officer  Representative:  The  COR  is  not  a  contracting  officer, 
nor  does  he/she  hold  any  contracting  authority.  The  FAR  specifically  states  that  a 
COR  “has  no  authority  to  make  any  commitments  or  changes  that  affect  price, 
quality,  quantity,  delivery,  or  other  terms  and  conditions  of  the  contract  nor  in  any 
way  direct  the  contractor  or  its  subcontractors  to  operate  in  conflict  with  the  contract 
terms  and  conditions”  (FAR  1 ,602-2(d)(5)).  The  function  of  a  COR  is  to  be  the  eyes 
and  ears  of  the  KO.  Ideally,  the  COR  would  be  a  technical  expert  and  assist  with 
monitoring  contractor  performance  on  a  specific  contract.  In  practice,  it  is  more 
likely  that  a  COR  will  not  have  a  high  degree  of  technical  expertise  in  the  field  of 
monitoring  contractor  performance. 

Functional  Area  Expert:  Functional  area  experts  have  a  specific  set  of  skills 
and  provide  the  acquisition  team  with  a  wide  range  of  talent  that  they  can  tailor  to 
each  service  contract.  They  provide  an  essential  service  by  reducing  the  workload 
and  technical  knowledge  required  by  the  contracting  officer.  Functional  area  experts 
include,  but  are  not  limited  to,  financial  managers  who  keep  the  project  on  budget, 
price/cost  estimators  who  analyze  the  price/cost  of  a  project,  legal  officers, 
engineers  who  have  technical  knowledge,  auditors,  small  business/competition 
advocates,  quality  assurance  representatives,  information  technology  advisors,  and 
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logisticians  (Beers,  2011).  Service  contracts  can  range  from  simple  janitorial 
services  to  highly  complex  multi-year  helicopter  maintenance  and  repair  services 
that  are  performed  on  multiple  bases  in  the  continental  United  States;  overseas 
bases  in  Europe,  Africa,  and  Korea;  and  warzones  like  Afghanistan  and  Iraq. 

Step  2:  Review  Current  Strategy 

Step  2  begins  with  the  acquisition  team  creating  a  baseline  and  then 
analyzing  the  current  service  strategy  (Beers,  201 1).  This  consists  of  looking  at 
historical  data,  identifying  problems,  projecting  any  type  of  modification  to  the 
mission,  and  getting  the  customer/end  user  to  define  the  key  performance  outcomes 
that  they  expect  from  this  contract.  This  is  by  far  the  most  important  part  of  Step  2. 
The  customer  must  be  able  to  define  what  they  expect  this  contract  to  accomplish  as 
there  can  be  differences  between  what  is  said  and  what  is  understood. 

For  example,  a  customer  requested  a  janitorial  cleaning  services  contract  to 
cover  portable  toilets  at  a  specific  location.  The  contracting  officer  understands  that 
this  contract  is  to  cover  cleaning  the  portable  toilets  and  removing  waste,  but  goes 
back  to  the  customer  to  define  key  requirements  to  make  sure  everyone  has  the 
same  view  of  the  performance  requirements.  The  customer  states  that  they  want 
the  vendor  to  check  the  level  of  toilet  paper  in  each  portable  toilet  and  resupply  each 
unit  as  necessary.  The  customer  also  requires  that  the  services  contract  cover  the 
removing  of  waste  water  and  trash  from  the  hand  washing  station  next  to  the 
portable  toilets  and  the  resupplying  of  the  hand  sanitizer  solution  and  paper  towels 
at  that  station.  What  the  customer  originally  stated  as  the  requirement  and  what 
he/she  thought  he/she  was  getting  was  different  from  what  the  contracting  officer 
understood  from  the  original  request.  This  is  why  it  is  important  for  the  customer 
and  acquisition  team  to  define  the  requirements  and  expected  performance 
outcomes. 

Step  3:  Market  Research 

As  Beers  (2011)  wrote,  “The  team  analyzes  the  marketplace  to  assess 
current  technology  and  business  practices,  competition  and  small  business 
opportunities,  and  existing  and  potential  new  sources  of  providing  the  service;  the 
team  then  determines  if  commercial  buying  practices  can  be  used”  (p.  56).  Ideally, 
the  customer  or  end  user  would  have  conducted  market  research  prior  to  giving  the 
requirements  to  the  contracting  command.  Market  research  can  provide  the 
customer  and  the  contracting  officer  with  ideas  about  the  types  of  services  required, 
possible  vendors  to  provide  those  services,  estimated  pricing  and  cost  data,  and 
help  defining  what  the  customer  really  needs.  One  of  the  main  reasons  for  doing 
market  research  is  to  determine  whether  a  service  is  commercially  available  or  is  a 
government-unique  service.  Commercial  services  tend  to  be  more  defined  and  have 
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more  vendors  to  compete  for  government  business.  Government-unique  services 
can  be  more  complicated  and  require  more  effort  to  define  the  requirement. 

Phase  2:  Develop 

The  second  phase  of  the  services  acquisition  process  is  the  Development 
phase.  This  phase  consists  of  Step  4,  Requirements  Definition,  and  Step  5, 
Acquisition  Strategy,  in  which  the  acquisition  team  develops  the  overall  plan  to 
procure  the  services.  The  requirements  and  performance  outcomes  that  the 
customer  provided  are  examined  and  refined  further.  Then  the  team  determines  the 
appropriate  path  to  contract  for  the  required  services. 

Step  4:  Requirements  Definition 

This  step  allows  the  acquisition  team  to  define  the  performance  objectives, 
determine  an  acceptable  variation,  and  choose  the  method  of  measuring 
performance  (Beers,  2011).  This  process  leads  the  team  to  develop  the 
Performance  Work  Statement  (PWS).  The  PWS  should  tie  directly  to  the 
requirements  and  is  used  to  define  how  the  vendor  will  be  evaluated  with  regard  to 
CPARS  objective  scores. 

The  following  is  an  example  of  defining  requirements.  The  Army  Mission  and 
Installation  Contracting  Command  (AMICC)  received  a  request  from  a  customer  to 
purchase  aircraft  maintenance  services.  This  is  poorly  defined  because  there  are 
many  different  types  of  aircraft  that  require  different  types  of  maintenance.  This 
could  mean  ordering  spare  parts,  engine  repair,  repair  of  proprietary  hardware, 
repair  of  electrical  or  electronic  systems,  or  some  other  type  of  maintenance.  The 
requirement  doesn’t  say  where  the  maintenance  services  will  be  performed.  When 
does  the  customer  need  the  services?  How  many  aircraft  will  be  serviced  or  how 
much  maintenance  service  does  the  customer  require?  The  requirement  provided  to 
the  contracting  official  is  very  vague,  which  doesn’t  allow  him/her  to  determine  what 
the  customer  wants  or  needs.  The  contracting  official  will  need  to  discuss  the 
requirements  with  the  customer  to  define  the  requirements. 

Step  5:  Acquisition  Strategy 

This  step  is  designed  to  select  the  contract  type  that  is  the  most  beneficial  to 
the  government  to  mitigate  risk,  provide  for  the  proper  amount  of  incentive  to  the 
vendor  to  complete  the  contract  objectives,  and  select  between  lowest  price  and 
trade-offs  where  the  best  value  may  be  a  higher  priced  solution  (Beers,  2011).  The 
requirements  will  tell  the  vendor  what  the  performance  metrics  are,  not  the  specific 
methods  for  how  the  vendor  will  complete  the  job.  The  customer  and  acquisition 
team  care  only  about  the  performance  outcomes,  not  how  the  vendor  accomplishes 
the  task.  This  allows  the  vendor  to  be  innovative  about  how  they  design  a  solution. 
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Phase  3:  Execute 


The  third  phase  of  the  services  acquisition  plan  is  the  Execution  phase.  This 
phase  consists  of  Step  6,  Execute  Strategy,  and  Step  7,  Performance  Management, 
which  pulls  together  everything  that  the  customer  and  acquisition  team  have 
developed  up  to  this  point  and  culminates  in  the  requirement  going  out  to  industry  in 
the  form  of  a  request  for  proposals  (RFP).  The  final  part  of  Phase  3  is  concerned 
with  contract  performance  and  administration  processes  designed  to  close  out  or 
terminate  a  contract. 

Step  6:  Execute  Strategy 

This  step  involves  the  creation  of  synopsis  and  solicitation  documents  that 
“formally  communicate  to  industry  the  customer’s  requirements  and  business  plan” 
(Beers,  201 1 ,  p.  57).  The  acquisition  team  may  release  a  draft  RFP  to  solicit 
industry  feedback  on  the  feasibility  of  the  requirements  and  to  get  a  feel  for  industry 
interest  in  performing  this  contract.  Once  the  requirement  is  ready  for  release  to  the 
public,  vendors  will  review  the  documents  and  create  a  solution  to  meet  those 
objectives.  The  acquisition  team  may  establish  a  competitive  range  to  limit  the 
number  of  applicants  to  an  economical,  efficient,  and  manageable  level  as  their  time 
and  resources  are  limited,  and  the  team  cannot  process  100,  50,  or  even  10 
proposals  if  the  requirements  are  complex. 

The  acquisition  team  will  evaluate  each  proposal  against  the  requirements 
specified  in  the  RFP  and  the  standards  and  performance  measures  included  in  the 
PWS.  The  team  will  not  evaluate  proposals  against  each  other.  The  acquisition 
team  will  recommend  to  the  source  selection  authority  (SSA)  the  proposal  that  best 
meets  the  RFP  requirements.  The  final  decision  on  selecting  the  best  proposal  rests 
with  the  SSA.  The  SSA  may  be  one  or  more  levels  above  the  contracting  officer. 

The  contracting  officer  will  then  inform  all  the  non-selects  of  the  results.  If  there  is 
some  discrepancy  with  the  contract  award,  the  non-selects  will  have  10  days  from 
the  contract  award  to  protest  the  award.  At  this  point,  the  acquisition  team  will  move 
into  the  administrative  or  management  step. 

Step  7:  Performance  Management 

As  Beers  (2011)  observed,  this  step  “involves  two  key  areas:  administering 
contract  requirements,  such  as  invoicing  and  payments,  and  managing  the 
relationships  and  expectations  of  both  the  contractor  and  customers  in  meeting  the 
terms  of  the  contract  and  achieving  the  required  mission  performance  results”  (p. 

57).  Many  in  the  contracting  world  argue  that  this  step  is  where  the  real  work 
begins. 
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Contractor  performance  on  the  contract  must  be  monitored,  which  is  where 
the  COR  aids  the  contracting  officer.  The  COR  will  communicate  with  the  contractor 
and  contracting  officer  regularly  to  provide  progress  reports  and  help  solve  problems 
as  they  arise.  Contracts  can  be  terminated  for  default  (T4D),  terminated  for 
convenience  (T4C)  of  the  government,  or  closed  out  upon  successful  completion. 
While  contracts  should  be  closed  out  within  six  months,  there  are  situations  in  which 
contracts  remain  open  for  decades  while  the  government  and  contractor  disagree 
over  payments,  terms  and  conditions,  and  settlement  of  any  court  appeals.  An 
example  is  Boeing  determining  what  the  final  labor  rate,  overhead  rate,  and  time  and 
material  rates  are  for  a  cost  reimbursement  type  contract.  Boeing  must  wait  until  all 
other  product  lines  have  provided  data  so  the  company  can  break  down  those  costs 
to  the  relevant  contract  and  develop  these  rates  for  reimbursement.  The 
government  will  then  review  those  costs  for  accuracy,  which  can  take  up  to  a  year  or 
more  on  complex  contracts.  If  there  is  a  discrepancy,  it  can  go  to  the  contract 
agency  head,  the  GAO,  or  the  Armed  Services  Board  of  Contract  Appeals,  which 
can  add  years  to  the  case.  The  contract  cannot  be  closed  out  until  every  last 
discrepancy  is  resolved,  even  if  it’s  as  simple  as  a  $0.01  difference  between  the 
contracted  price  and  the  contractor  invoice. 

Contractor  performance  must  be  monitored  regularly  to  comply  with  contract 
administration  requirements.  Contract  administration  can  cover  short  periods  of  time 
or  govern  multiple  periods,  depending  on  the  type  of  contract.  A  contract  with  a 
short  period  of  performance  may  see  the  source  selection,  contract  award,  contract 
performance,  and  closed  out  periods  all  occur  within  the  same  fiscal  year.  In  this 
instance,  a  contracting  officer  would  provide  the  CPARS  with  an  evaluation  of  that 
contractor’s  performance  on  that  particular  contract  based  upon  the  evaluation 
criteria  set  forth  in  the  contract.  A  contract  that  crosses  fiscal  years,  has  multiple 
option  years  of  the  same  contract,  or  is  a  multi-year  contract  (e.g.,  ship  building, 
military  construction),  may  require  a  CPARS  evaluation  report  on  the  contractor’s 
performance  on  an  annual  or  more  regularly  scheduled  basis  and  a  final  CPARS 
evaluation  when  the  contract  closes  out.  Military  personnel  move  on  a  regular  basis 
so  the  PM,  KO,  or  COR  who  starts  on  a  contract  source  selection  may  be  different 
from  the  PM,  KO,  or  COR  who  administers  the  contract,  who  might  differ  from  the 
PM,  KO,  or  COR  who  closes  out  the  contract.  Diligent  monitoring  of  contractor 
performance  is  the  only  way  to  accurately  evaluate  and  rate  a  contractor  in  CPARS 
over  the  life  of  the  contract. 

Accurate  CPARS  report  cards,  objective  scores,  and  descriptive  narratives 
are  important  because  they  are  used  during  the  government’s  source  selection 
process.  As  described  in  Figure  1,  the  Services  Acquisition  Plan,  the  government 
will  execute  the  acquisition  strategy  by  selecting  a  contractor  and  monitoring 
performance.  Contractor  past  performance  can  play  a  significant  role  in  determining 
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eligibility  and  competitiveness  in  the  source  selection  process.  Contractor  past 
performance  can  demonstrate,  but  is  not  limited  to,  contractor  responsiveness  to 
customer  needs,  reliability,  contractor’s  past  performance  on  relevant  contracts, 
government/contractor  business  relationships,  disqualifying  conditions  (debarment, 
suspension,  termination  for  default,  failure  to  meet  standards),  contractor/ 
subcontractor  business  relationships,  and  factors  to  mitigate  potential  risk.  Given 
the  role  that  past  performance  information  plays  in  the  source  selection  process,  it  is 
important  to  review  the  CPARS  system.  The  next  section  provides  an  overview  of 
CPARS,  the  narrative  and  issues  surrounding  CPARS  information. 

C.  CONTRACTOR  PERFORMANCE  ASSESSMENT  REPORTING  SYSTEM 

The  CPARS  is  used  by  all  federal  agencies  to  evaluate  contractor 
performance.  The  purpose  behind  this  system  is  to  give  federal  procurement 
personnel  a  way  to  assess  “a  contractor’s  performance,  both  positive  and  negative, 
and  provide  a  record  on  a  given  contract  during  a  specified  period  of  time”  (Hart  et 
al.,  2013,  p.  20).  CPARS  report  cards  are  segmented  into  five  distinct  business 
sectors:  Systems,  Services,  Operations  Support,  Fuels,  and  Information  Technology 
(DoN,  1997).  The  key  business  sector  “Services”  can  be  subdivided  into 
Professional/Technical  &  Management  Support  Services,  Repair  &  Overhaul,  and 
Installation  Services.  The  following  excerpt  is  from  the  CPARS  manual  and 
describes  the  types  of  contracts  that  are  considered  “Services”  contracts. 

Services:  Generally,  all  contracted  services  except  those  related  to 
“Science  &  Technology,”  “Construction  &  Architect — Engineering 
Services,”  and  “Health  Care.” 

Professional/Technical  &  Management  Support  Services:  Includes 
all  consultant  services — those  related  to  scientific  and  technical 
matters  (e.g.,  engineering,  computer  software  engineering  and 
development),  as  well  as  those  related  to  organizational  structure, 
human  relations,  etc.  Includes  office  administrative  support  services 
(e.g.,  any  basic  or  applied  research  that  will  result  in  new  or  original 
works,  concepts,  or  applications,  but  does  not  include  contract  advice 
on  the  feasibility  of  such  research,  as  well  as  evaluation  of  research 
results). 

Repair  &  Overhaul:  Services  related  to  the  physical  repair  and 
overhaul  of  aircraft  ground  vehicles,  etc.,  and  any  associated 
subsystems  or  components.  Includes  condition  evaluations  of 
individual  items  received  for  repair  or  overhaul,  but  does  not  include 
evaluations  of  the  feasibility  or  the  benefits  of  the  overall  project.  Does 
not  include  ship  repair  and  overhaul  which  is  included  under  the 
Systems  sub-sector  on  Shipbuilding. 

Installation  Services:  Includes  services  for  grounds  maintenance 
(grass  cutting,  shrubbery  maintenance  or  replacement,  etc.).  Includes 
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services  related  to  cleaning,  painting,  and  making  minor  repairs  to 
buildings  and  utilities  services,  etc.  Includes  contracted  security  and 
guard  services.  Includes  installation  and  maintenance  of  fencing. 
Includes  minor  electrical  repairs  (e.g.,  replacing  outlets,  changing  light 
bulbs,  etc.).  Includes  minor  road  surface  repairs  (patching  cracks, 
filling  in  potholes,  etc.).  Includes  relocation  of  individual  telephone 
lines  and  connections.  Includes  snow  removal.  (But,  see  also 
“Construction  &  Architect/Engineering  Services”  and  “Information 
Technology”  for  the  services  covered  by  those  business  areas.).  (DoN, 
1997,  p.  A1-2) 

The  Services  Business  Sector  is  a  very  diverse  collection  of  services. 
Contracting  professionals  may  not  be  familiar  or  experienced  with  such  a  wide  array 
of  contracted  services  so  it  is  important  for  CORs,  KOs,  PMs,  and  the  contract 
management  team  to  work  together  to  ensure  that  the  contractor’s  performance  is 
clearly  and  accurately  recorded  in  the  CPARS  report  card. 

Past  performance  information  entered  into  CPARS  must  be  accurate  and 
timely  as  it  is  primarily  used  by  source  selection  officials  to  award  or  not  award  a 
contract  based  on  a  contractor’s  strengths  and  weaknesses  during  Step  6,  the 
Execute  Strategy  phase  of  the  Services  Acquisition  Plan.  As  explained  in  the 
CPARS  manual,  “The  value  of  CPARS  to  a  future  source  selection  team  is 
inextricably  linked  to  the  care  the  program  manager  takes  in  preparing  a  quality 
narrative  to  accompany  the  CPAR  scores”  (DoN,  1997,  p.  1). 

The  CPARS  evaluation  assesses  at  least  five  distinct  areas  of  contractor 
performance  supported  by  objective  data  presented  in  Block  20  (DoN,  1997).  These 
areas  are  Quality  of  Product/Service,  Schedule,  Cost  Control,  Business  Relations, 
and  Management  of  Key  Personnel.  Objective  scores  are  used  by  the  acquisition 
team  to  rate  contractor  performance  in  these  five  areas.  There  are  five  possible 
scores  for  each  of  these  five  areas.  The  five  possible  scores  are  unsatisfactory, 
marginal,  satisfactory,  very  good,  and  exceptional  (also  shown  in  Table  3). 

CPARS  includes  a  variance  assessment  that  assesses  the  variance  between 
current  costs  and  schedules,  and  government  estimates.  Scoring  for  Block  18  and 
19  must  be  in  accordance  with  Appendix  B,  Specific  Examples  Of  Narrative 
Statements  To  Avoid  From  The  CPAR  Quality  Checklist,  and  Evaluation  Ratings 
Definitions  (DoD,  2011,  pA2-1 )),  and  is  described  in  detail  in  Chapter  III — Research 
Methodology. 

Finally,  CPARS  includes  a  program  manager  narrative  that  is  a  statement  of 
the  facts  regarding  the  contractor’s  performance  on  a  particular  contract.  The 
program  manager  is  responsible  for  generation  and  content  of  the  narrative  as  the 
PM  must  sign  and  date  this  form  prior  to  sending  it  to  the  KO  for  review  (DoN,  1997). 
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The  narrative  comments  in  Block  20  should  directly  trace  back  to  the  corresponding 
objective  scores  in  Blocks  18  and  19  of  the  CPARS  report  card  (DoN,  1997). 

Consistent  and  objective  evaluations  of  contractor  performance  are  essential, 
which  is  why  a  series  of  checks  and  balances  was  designed  within  the  CPARS 
process  (Hart  et  al.,  2013).  As  stated  in  the  CPARS  manual,  “Each  assessment 
must  be  based  on  objective  facts  and  be  supportable  by  program  and  contract 
management  data,  such  as  cost  performance  reports,  customer  comments,  quality 
reviews,  technical  interchange  meetings,  financial  solvency  assessments,  and 
functional  performance  evaluations”  (DoN,  1997,  p.  2). 

Before  the  performance  assessment  can  be  finalized,  it  must  be  forwarded  to 
the  contractor  for  review  and  comment.  This  ensures  the  past  performance 
information  report  card  is  finalized  only  after  the  contractor  has  the  opportunity  to 
provide  feedback  to  the  assessing  official.  The  contractor  is  not  required  to  respond 
but  must  have  the  opportunity  to  provide  inputs  to  the  CPARS  evaluation.  Only  after 
this  has  happened  can  the  approving  official  finalize  the  report.  The  approving 
official  is  someone  in  a  position  at  least  one  level  above  the  program  manager  to 
help  ensure  impartiality  and  objectivity  (DoN,  1997).  Even  with  these  checks  and 
balances  in  place,  the  Army  and  the  DoD  as  a  whole  have  had  below-standards 
records  for  CPARS  compliance.  Table  1  shows  the  compliance  rates  for  the  Army 
and  the  larger  DoD  with  a  49%  and  56%,  respectively,  in  201 1  but  much  improved 
compliance  rates  of  71%  and  74%,  respectively,  for  2013  (GAO,  2013,  p.  9). 

Table  1.  Percentage  of  Required  Assessments  Submitted  to  the  Past 
Performance  Information  Retrieval  System  (PPIRS) 

(GAO,  2013,  p.  9) 


DoD  Compliance  Rate  as  of 


Component 

2011 -Oct-03 

2012-Sep-28 

2013-Apr-OI 

Air  Force 

82% 

82% 

80% 

Navy 

66% 

69% 

72% 

Army 

49% 

60% 

73% 

Other  DoD 

32% 

61% 

71% 

Total  DoD 

56% 

66% 

74% 

The  following  section  addresses  several  issues  identified  by  the  GAO  and  the 
Office  of  Federal  Procurement  Policy  (OFPP)  in  the  area  of  documenting  contractor 
past  performance  information  in  CPARS  and  their  steps  to  correct  and  improve  the 
process. 

D.  INCREASED  SCRUTINY  OF  FEDERAL  CONTRACTING  AGENCIES 

An  increased  focus  on  contractor  past  performance  began  in  1994  with  the 
Federal  Acquisition  Streamlining  Act  (FASA),  in  which  Congress  directed  that 
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government  contracting  agencies  should  use  contractor  past  performance 
information  as  one  of  the  award  decision  evaluation  factors.  As  a  result  of  the 
FASA,  the  FAR  was  updated  to  reflect  this  new  requirement,  and  the  OFPP  began 
releasing  guides  and  memoranda  for  the  purpose  of  instructing  federal  contracting 
professionals  in  the  best  practices  for  collecting,  submitting,  and  utilizing  contractor 
past  performance  information. 

One  of  the  biggest  obstacles  in  the  way  of  achieving  more  reliably  widespread 
and  frequent  use  of  contractor  past  performance  information  in  the  contracting 
process  was  the  lack  of  a  centralized  reporting  system.  In  2002,  the  DoD  directed 
that  the  PPIRS,  a  web-enabled  contractor  past  performance  information  database 
and  retrieval  tool,  was  its  single  authorized  retrieval  system  for  past  performance 
information.  However,  a  GAO  report  from  2009  revealed  that  “PPIRS  data  for  fiscal 
years  2006  and  2007  indicates  that  only  a  small  percentage  of  contracts  had  a 
documented  performance  assessment”  (GAO,  2009,  Executive  Summary).  Table  2 
shows  the  disparity  between  the  total  numbers  of  contracts  requiring  assessments 
versus  contracts  with  an  actual  completed  assessment.  In  2007,  the  Air  Force  led 
the  way  with  47%.  In  2009,  the  FAR  began  requiring  all  federal  agencies  to  submit 
all  of  their  contractor  performance  evaluations  to  PPIRS. 

Table  2.  DoD  CPARS  Requirements  vs.  Actual  Reporting 

(GAO,  2009,  p.  12) 


Estimated  Contracts  Requiring  an  Assessment  That  Had  an  Assessment  In  Fiscal  Year  2007 

Estimated  contracts 

Contracts  with  an 

Department/agency 

requiring  an  assessment 

assessment 

Percent 

Air  Force 

2,795 

1,300 

47 

Navy 

3,879 

1,622 

42 

Army 

6,145 

1,971 

32 

Other  DOD 

1,408 

303 

22 

Homeland  Security 

4,131 

535 

13 

NASA 

3,706 

1,093 

29 

Energy 

840 

183 

22 

Total  percentage 

22.904 

7.007 

31 

Source.  GAO  analysis  of  data  from  DOO.  FPDS-NG.  and  PPIRS. 


In  2007,  the  GAO  released  a  report,  Use  of  Contractor  Performance 
Information,  detailing  its  study  of  how  contractor  past  performance  information  may 
be  considered  in  the  contracting  process  and  what  issues  government  contracting 
agencies  have  encountered  with  the  use  of  this  information.  The  report  identifies 
many  of  the  complexities  that  go  into  the  process  of  evaluating  contractor  past 
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performance  and  how  these  complexities  have  affected  GAO  protest  decisions 
(GAO,  2007). 

Contractor  past  performance  information  is  fundamentally  important  to  the 
contracting  process  from  the  beginning  during  the  pre-award  period  through  the  end 
of  contract  performance.  The  GAO  report  looked  at  several  ways  in  which 
government  contracting  finds  use  in  considering  contractor  past  performance. 

These  include  source  selection,  responsibility  determinations,  surveillance  of 
performance  under  the  current  contract,  and  suspension  and  debarment  (GAO, 
2007). 

A  GAO  report  from  2009  revealed  the  results  of  a  review  of  62  government 
contract  solicitations  from  fiscal  years  2007  and  2008  for  which  contractors’  past 
performance  information  was  considered  (GAO,  2009).  The  researchers  also 
interviewed  121  contracting  officials,  asking  them  about  how  their  agencies  utilize 
contractor  past  performance  information.  The  GAO  study  showed  that  although 
contractor  past  performance  information  was  being  reviewed,  to  include  the  use  of 
the  PPIRS  system,  “factors  other  than  past  performance,  such  as  technical 
approach  or  cost,  were  the  primary  factors  for  contract  award  decisions”  (GAO, 

2009,  Executive  Summary).  While  the  FAR  does  allow  contracting  officials  broad 
discretion  for  how  heavily  to  weight  contractor  past  performance  relative  to  other 
evaluation  factors  in  their  source  selection  evaluation,  the  GAO  study  revealed  that 
“a  majority  of  officials  told  us  their  reluctance  to  rely  more  on  past  performance  was 
due,  in  part,  to  their  skepticism  about  the  reliability  of  the  information  and  difficulty 
assessing  relevance  to  specific  acquisitions”  (GAO,  2009,  p.  8). 

The  OFPP  followed  the  GAO  report  in  2009  with  a  memorandum  for  the  chief 
acquisition  officers  and  senior  procurement  executives  which  acknowledged  the 
“fragmented  methods”  that  agencies  used  to  collect  and  maintain  contractor 
performance  information.  The  OFPP  stated  that  agencies  “maintain  evaluations  in 
internal  data  systems  that  are  not  available  to  acquisition  officers  outside  that 
agency”  (OFPP,  2009,  p.  1 ).  This  was  occurring  despite  the  FAR  requirement  that 
contractor  performance  information  be  shared  between  agencies.  The  OFPP  issued 
new  requirements  to  agencies  pursuant  to  FAR  subpart  42.15  changes  effective  July 
2009  mandating  submission  of  contractor  performance  records  to  PPIRS.  The 
OFPP  encouraged  government  contracting  agencies  to  designate  specific 
individuals  to  the  task  of  ensuring  that  “accurate,  complete,  and  timely  information  is 
submitted  to  PPIRS”  (OFPP,  2009,  p.  2). 

In  2011,  the  OFPP  issued  a  memorandum  to  share  the  findings  of  its  review 
of  agency  compliance  with  its  2009  directives  and  to  provide  additional 
recommendations  for  continued  improvement  in  the  collection  of  contractor  past 
performance  information.  Researchers  looked  at  nearly  700  performance  reports 
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submitted  by  the  10  federal  contracting  agencies  that  make  up  the  bulk  of  the  federal 
contracting,  together  accounting  for  94%  of  federal  contract  obligations  for  fiscal 
year  2009  (OFPP,  2011).  This  review  took  place  during  the  period  in  which 
contracting  agencies  were  migrating  to  the  Contractor  CPARS  for  the  purpose  of 
submitting  their  contractor  performance  information  to  PPIRS. 

The  OFPP  found  that  in  2009,  the  DoD  had  conducted  required  past 
performance  evaluations  on  about  50%  of  contract  awards  and  that  the  quality  of 
those  submitted  were  of  varying  quality  with  most  deemed  lacking  in  sufficient 
information.  They  evaluated  the  contractor  performance  report  narratives  based  on 
how  well  they  addressed  the  four  required  rating  factors:  quality  of  the  product  or 
service,  ability  to  control  cost,  ability  to  meet  schedule,  and  quality  of  business 
relations  (OFPP,  2011).  The  DoD  has  used  CPARS  for  submitting  contractor  past 
performance  reports  since  2004  (GAO,  2013). 

The  OFPP  review  of  performance  reports  submitted  by  the  DoD  revealed  the 
following:  53.1%  contained  sufficient  narrative  for  quality  of  product/service,  51% 
contained  sufficient  narrative  for  schedule  control,  21.9%  contained  sufficient 
narrative  for  cost  control,  and  50%  contained  sufficient  narrative  for  business 
relations  (OFPP,  2011). 

The  OFPP  issued  guidance  to  improve  the  quality  of  contractor  past 
performance  reporting.  Agencies  were  directed  to  establish  roles  and 
responsibilities  for  those  individuals  appointed  to  review  and  submit  performance 
reports  and  to  facilitate  proper  training  for  all  acquisitions  personnel.  A  new 
increased  focus  on  training  and  accountability  was  expected  to  significantly  improve 
contractor  performance  reporting.  Agencies  are  required  to  establish  an  agency 
point  of  contact  accountable  for  disseminating  guidance,  facilitating  training, 
developing  oversight,  and  identifying  improvements  to  the  submission  process. 

They  were  directed  to  institute  a  performance  report  review  process  for  the  purpose 
of  monitoring  their  quality  (OFPP,  2011). 

In  2013,  the  OFPP  took  additional  steps  to  improve  the  quality  of  contractor 
performance  assessments.  In  March,  it  released  a  memorandum  establishing  a 
baseline  for  compliance  with  contractor  performance  reporting  standards  and  setting 
agency  performance  targets  for  fiscal  years  2013  through  2015  (OFPP,  2013). 

A  2013  GAO  report  shows  that  measures  taken  by  the  OFPP  and  the  FAR 
Council  have  led  to  higher  rates  of  submission  for  performance  assessments  within 
the  DoD.  In  2007,  fewer  than  half  were  completed.  Submission  of  required 
assessments  increased  from  56%  in  October  2011  to  74%  by  April  2013.  However, 
the  report  shows  that  the  DoD  is  still  failing  to  complete  assessments  on  time.  As 
Figure  2  shows,  a  large  portion  of  the  required  assessments  are  completed  late  (i.e., 
after  the  120-day  requirement;  GAO,  2013). 
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Figure  2.  DoD  CPARS  Reporting  Compliance  2010-2012 

(GAO,  2013,  p.  10) 


The  GAO  report  focuses  on  quantifying  the  effectiveness  of  the  measures 
taken  by  the  DoD  to  improve  the  quality  and  timeliness  of  its  contractor  performance 
reports  that  it  submitted  via  CPARS.  DoD  requirements  for  contractor  performance 
reports  (as  specified  in  the  CPARS  Guide)  specify  that  performance  assessments 
must  be  completed  and  finalized  within  120  days  from  the  end  of  the  contractor 
evaluation  period  and  that  the  assessing  official  is  responsible  for  preparing  and 
finalizing  the  report.  The  report  indicates  that 

although  the  CPARS  Guide  does  not  currently  specify  standards  for 
completeness  ...  a  recent  proposed  change  to  the  FAR  will  address 
completeness  by  providing  minimum  government-wide  standards  for 
past  performance  rating  elements.  Specifically,  the  proposed  rule 
requires  that  all  assessments  address,  at  a  minimum,  quality  of 
product  or  service,  timeliness,  and  management  or  business  relations. 
(GAO,  2013,  p.  8) 


As  of  June  2013,  there  were  no  formally  established  standards  for  quality  and 
completeness  of  CPARS  performance  narratives. 

CPARS  data  are  not  and  should  not  be  the  only  source  of  information  used  to 
determine  contractor  past  performance  (DoD,  2012).  Other  sources  of  PPI  are 
references  provided  by  said  contractor,  references  provided  by  other  federal 
contracting  personnel,  and  other  information  not  in  CPARS.  One  of  the  limitations  of 
CPARS  is  that  federal  contracting  personnel  are  not  required  to  enter  PPI  into 
CPARS  if  the  dollar  amount  of  the  specific  contract  does  not  meet  the  minimum 
reporting  threshold,  as  described  in  Figure  3.  While  the  government  may  elect  to 
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enter  PPI  into  CPARS  below  these  thresholds,  this  is  not  common  practice  and,  as  a 
result,  leads  to  gaps  in  PPI  data  below  the  $1  million  threshold  for  service  contracts. 


Business  Sector 

Dollar  Threshold 

Systems 

>  $5,000,000 

Ship  Repair  &  Overhaul  >  $500,000 

Services 

>$1,000,000 

Health  Care 

>  $150,000 

Operations  Support 

>  $5,000,000 

Fuels 

>  $150,000 

Information  Technology  >  $1,000,000 

Figure  3.  DoD  Reporting  Thresholds 


E.  PREVIOUS  STUDIES 

This  is  the  seventh  study  of  a  series  of  research  projects  aimed  at  services 
contract  management.  The  original  research  began  in  2006  when  Apte,  Ferrer, 
Lewis,  and  Rendon  examined  a  growing  trend  of  DoD  acquisition  workload  over  the 
previous  decade.  As  noted  by  Hart  et  al.  (2013),  one  of  the  major  observations  of 
this  research  was  a  66%  increase  in  services  contracting  since  1999  but  no 
corresponding  increase  in  contract  management  personnel  (Apte  et  al.,  2006). 

The  second  research  project  was  an  exploration  of  supply  chain  management 
of  service  contracts  (Apte  &  Rendon,  2007).  In  general  terms,  the  research  focused 
upon  how  the  Air  Force  used  the  Air  Education  and  Training  Center  (AETC)  and 
Acquisition  Management  and  Integration  Center  (AMIC)  models  (Hart  et  al.,  2013). 
Apte  and  Rendon  observed  that  these  processes  enabled  cradle-to-grave  handling 
of  a  service  acquisition  but  allowed  for  a  communication  failure  as  program 
managers  are  not  on  site  (Hart  et  al.,  2013). 

The  third  research  project  differed  from  the  previous  ones  in  that  this  was 
based  on  a  survey  of  Air  Force  and  Navy  service  contracts  (Hart  et  al.,  2013).  Apte, 
Apte,  and  Rendon  (2008)  focused  upon  program  management  issues  and 
approaches  of  service  contracting.  This  was  relevant  to  research  conducted  by  Hart 
et  al.(2013)  but  is  not  directly  relevant  to  the  narrative  evaluation  in  CPARS. 

The  fourth  research  project  focused  on  services  contract  management  was 
published  in  2009  and  expanded  upon  the  third  research  project  (Apte  et  al.,  2008) 
by  including  the  Army  in  the  survey.  Again,  problems  were  noted  with  the 
acquisition  team  and  communication  issues  (Hart  et  al.,  2013).  The  major  takeaway 
from  this  research  is  that  most  acquisition  professionals  believed  that  “their 
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organizations  [did  not  have]  sufficient  positions  and  that  the  available  positions  were 
[not]  adequately  filled”  (Hart  et  al.,  2013,  p.  29).  Like  the  previous  research  projects, 
this  improves  the  knowledge  base  for  service  contracts  but  is  not  directly  related  to 
the  narrative  evaluation  in  CPARS. 

The  fifth  research  project,  conducted  by  Apte,  Apte,  and  Rendon  in  2010, 
collected  data  from  the  previous  studies  and  analyzed  them  for  DoD  management 
techniques  for  services  acquisition  (Hart  et  al.,  2013).  They  focused  upon  improving 
the  communication  of  contracting  personnel  in  the  Army,  Air  Force,  and  Navy.  Like 
the  previous  research  projects,  this  improves  the  knowledge  base  for  service 
contracts  but  is  not  directly  related  to  the  narrative  evaluation  in  CPARS. 

The  sixth  research  project  was  completed  in  2012  by  Apte,  Apte,  and 
Rendon.  This  project  focused  upon  the  drivers  of  management  practices  of 
successful  service  contracts  in  the  Army  (Hart  et  al.,  2013).  Hagan,  Spede,  and 
Sutton  surveyed  168  key  personnel  to  determine  the  definition  of  a  successful 
service  contract  in  the  Navy  (Hart  et  al.,  2013).  The  outcome  of  this  research  was  a 
correlation  of  modifications,  protests,  and  communications  between  the  acquisition 
team  members.  Like  the  previous  research  projects,  this  improves  the  knowledge 
base  for  service  contracts  but  is  not  directly  related  to  the  narrative  evaluation  in 
CPARS. 

The  seventh  and  most  germane  research  project  was  completed  in  2013  by 
Hart,  Stover,  and  Wilhite.  The  group  explored  the  definition  of  a  successful  service 
contract  by  looking  at  the  CPARS  report  cards  stored  within  the  PPIRS.  A  local 
database  was  created  from  715  Army  service  contract  CPARS  report  cards  for  the 
purpose  of  evaluating  success  determined  by  six  CPARS  evaluation  areas:  Quality 
of  the  Product/Service,  Schedule,  Cost  Control,  Business  Relations,  Management  of 
Key  Personnel,  and  Utilization  of  Small  Business  (Hart  et  al.,  2013). 

Contractor  evaluation  scores  for  the  six  evaluation  areas  were  assigned  a 
numerical  value  by  the  researchers  according  to  the  adjectival  rating  scale  in  Table 
3.  The  research  calculated  the  average  scores  to  find  any  correlation  to  success  or 
failure  for  the  six  evaluation  areas.  This  rating  system  and  the  associated  database 
are  the  foundation  for  the  research  that  is  to  be  examined  in  Chapters  III  and  IV  of 
this  research  project. 
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Table  3.  Area  of  Contract  Evaluation  Scores 

(Hart  et  al. ,  2013,  p.  44) 


Rating 

Score 

Exceptional 

5 

Very  Good 

4 

Satisfactory 

3 

Marginal 

2 

Unsatisfactory 

1 

Of  the  715  Army  service  contracts  reviewed,  22  were  deemed  failures 
because  they  received  a  rating  of  unsatisfactory  or  marginal  for  one  or  more  of  the 
six  evaluation  areas  previously  listed  (Hart  et  al.,  2013).  Table  4  shows  the 
breakdown  of  the  715  contracts  reviewed  by  Hart  et  al.  (2013). 


Table  4.  Total  Contract  Information 

(Hart  et  al.,  2013,  p.  44) 


Failures 

Success 

Total 

Failure  Rate 

Contracts 

22 

693 

715 

3.08% 

Hart  et  al.  (2013)  found  that  service  contracts  had  an  average  failure  rate  of 
approximately  3%  (see  Table  4).  They  cited  several  factors  that  led  to  contract 
failure,  but  a  common  theme  among  them  was  a  lack  of  proper  training  for  source 
selection  evaluators  (SSEs).  SSEs  must  be  able  to  properly  evaluate  each 
contractor’s  proposal  according  to  the  requirements  in  the  RFP  during  the  source 
selection  process.  Past  performance  information,  to  include  objective  ratings  and 
descriptive  narratives,  plays  an  important  role  in  the  SSE’s  evaluation  of  the 
contractor’s  proposal.  It  is  this  past  performance  information  that  is  the  focus  of  our 
research  project. 

The  seven  previous  service  contract-related  research  projects  have  provided 
the  background  and  the  insight  to  pursue  this  research  project.  CPARS  report 
cards,  PPI,  and  the  database  of  715  Army  service  contracts  generated  by  Hart  et  al. 
(2013)  form  the  core  of  the  analysis  of  program  manager  narratives  and  their  relation 
to  the  objective  scores  in  CPARS. 

F.  SUMMARY 

This  chapter  presented  an  overview  of  the  DoD  services  acquisition  process 
and  the  literature  associated  with  this  process.  The  chapter  began  with  a 
presentation  of  the  service  contract  management  process  as  outlined  in  Beers’  201 1 
Services  Acquisition  Plan.  Next,  the  chapter  reviewed  the  CPARS  to  provide  a 
framework  for  the  database  which  is  the  basis  for  this  research  project.  Finally,  the 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  25  - 

NAVAL  POSTGRADUATE  SCHOOL 


chapter  concluded  with  a  review  of  previous  research  projects  related  to  the  service 
contracting  field  of  study. 

The  next  chapter  discusses  the  research  questions  that  this  research 
attempts  to  answer,  the  methods  used  to  conduct  the  research,  the  raw  data  to  be 
analyzed,  and  the  type  of  statistical  analysis  used  to  understand  and  explain  the 
relationships  in  the  data. 
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III.  RESEARCH  METHODOLOGY 


A.  INTRODUCTION 

This  chapter  discusses  how  the  research  was  conducted  and  how  the  authors 
answered  the  research  questions.  The  chapter  begins  by  describing  how  the 
objective  scores  and  narratives  are  entered  into  CPARS  and  how  it  relates  to  a 
successful  contract.  Then  the  chapter  contains  an  in-depth  description  of  how  those 
data  are  analyzed,  including  a  description  of  the  statistical  methodology  used  to 
perform  the  analysis.  The  last  part  of  this  chapter  describes  the  interviews  that  were 
conducted,  along  with  the  interview  question  development  methodology. 

This  research  examines  the  value  of  CPARS  report  card  narratives  for  service 
contracts  as  they  relate  to  their  associated  objective  scores.  The  methodology 
includes  data  analysis  (including  both  sentiment  and  statistical  analysis)  and 
interviews  with  government  agency  contracting  professionals. 

B.  OBJECTIVE  SCORES  AND  NARRATIVES 

The  first  phase  of  data  analysis  performed  was  a  sentiment  analysis  of  715 
Army  service  contract  CPARS  report  card  narratives.  The  CPAR  Quality  Checklist 
was  used  as  a  basis  for  developing  the  criteria  for  the  categories  and  values  for  the 
analysis  (“CPAR  Quality  Checklist,”  n.d.).  Data  from  715  CPARS  report  cards  of 
service  contracts  from  2012  were  made  available  as  the  basis  of  this  continuing 
research  (Hart  et  al .,  2013).  Each  sample  provided  a  series  of  objective  scores  in  a 
spreadsheet  that  was  associated  with  each  narrative. 

Table  5  provides  an  example  of  objective  scores  for  the  six  evaluation  areas 
used  to  rate  a  contractor’s  performance. 


Table  5.  Example  of  Objective  Scores 


Quality  of 
Product  or 

Service 

Schedule 

Cost 

Control 

Business 

Relations 

Management 
of  Key 
Personnel 

Utilization 
.  _  ..  Min 

of  Small 

_  .  Score 

Business 

Success 

or  Failure 

VERY 

VERY 

VERY 

VERY 

VERY 

VERY 

GOOD 

GOOD 

GOOD 

GOOD 

GOOD 

GOOD  4 

SUCCESS 

(4) 

(4) 

(4) 

(4) 

(4) 

(4) 

Table  6  provides  examples  of  narratives  that  describe  in  the  appropriate 
amount  of  detail  the  performance  of  the  contractor  for  that  particular  objective  score. 
For  example,  an  objective  score  of  very  good  for  the  area  of  quality  of  product  or 
service  does  not  tell  the  full  story.  The  narrative  associated  with  an  objective  score 
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of  very  good  is  meant  to  explain  how  the  contractor  maintained  a  very  good  level  of 
quality  via  a  quality  control  program  that  was  operated  efficiently  and  effectively 
throughout  the  life  of  the  contract. 

_ Table  6.  Example  of  Narratives  Associated  With  Objective  Scores 

Quality  of  Product/Service:  This  contractor  has  done  a  very  good  job  overall.  This  particular  task 
order  is  for  the  management  Firm  Fixed  Priced  piece  in  support  of  a  major  customer.  Overall,  quality 
of  the  products  and  services  remains  high  and  the  contractor  quality  control  program  runs  efficiently 
and  effectively. 

Schedule:  Contractor  continues  to  provide  products/services  on  time.  Contractor  continues  to  be 
very  adaptive  to  unanticipated  work  requests. 

Cost  Control:  Contractor  has  historically  produced  accurate  invoices.  This  is  certified  annually  under 
the  Contractor  Manpower  Reporting  system. 

Business  Relations:  Contractor  has  very  good  Business  Relations.  The  government  has  never 
encountered  any  difficulty  in  working  with  this  contractor  to  resolve  any  issue.  Contractor  has  three 
(3)  subcontractors  assisting  in  the  performance  of  this  operation.  Contractor  provided  daily  reports  on 
all  equipment  under  their  control  for  maintenance.  The  contractor  was  effective  in  programs  such  as 
equal  opportunity,  employee  incentive,  energy  conservation,  safety,  security,  and  upward  mobility. 
Management  of  Key  Personnel:  All  key  personnel  have  performed  very  well. 

Utilization  of  Small  Business:  Contractor  utilizes  three  small  businesses  as  subcontractors  on  this 
task  order. 

Overall  Comments:  Contractor  continues  to  satisfy  customers.  The  government  frequently  receives 
positive  comments  on  the  end  product  received  from  this  contractor.  Contractor  provides  a  very  good 
service  through  the  cost  plus  contract.  The  government  definitely  would  recommend  awarding  future 
contracts  to  this  contractor. 

Given  what  I  know  today  about  the  contractor's  ability  to  execute  what  he  promised  in  his  proposal,  I 
DEFINITELY  WOULD  award  to  him  today  given  that  I  had  a  choice. 


To  analyze  the  narratives  in  the  samples  and  see  how  they  relate  to  the 
objective  scores,  a  series  of  criteria  had  to  be  developed.  The  CPAR  Quality 
Checklist  was  used  as  a  basis  for  developing  the  criteria  (“CPAR  Quality  Checklist,” 
n.d.).  Narratives  were  assigned  several  scores  relating  to  their  quality,  robustness, 
compliance  with  directions  in  the  CPAR  Quality  Checklist,  and  value  and  content 
compared  to  their  related  objective  scores  from  the  CPARS  report  cards. 

First,  we  answered  a  series  of  yes  or  no  questions.  These  questions  are 
listed  as  follows. 

First,  does  the  narrative  address  all  performance  areas  assessed  as  objective 
scores?  This  means  that  if  one  of  the  five  objective  scores  was  marked  as  not 
applicable,  there  would  not  need  to  be  something  written  in  the  narrative  specifically 
mentioning  that  area.  Otherwise,  there  should  be  something  written  in  the  narrative 
to  address  the  score  given  for  each  area  in  the  objective  scores. 
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Second,  is  the  narrative  based  on  objective  data?  In  answering  this  question, 
objective  data  is  defined  as  not  being  influenced  by  personal  feelings  or  opinions  of 
the  person  inputting  the  data,  but  only  considering  and  representing  facts. 

Third,  is  the  narrative  free  of  statements  to  avoid,  as  defined  by  the  CPAR 
Quality  Checklist?  The  following  phrases  (or  those  closely  similar)  qualify  as 
narrative  statements  to  avoid:  “outside  the  scope  of  the  contract,”  “in  our  opinion,” 
“appeared,”  “we  believe,”  “it  is  our  hope,”  “we  are  not  happy,”  “we  did  not  like,”  “we 
think,”  “could  be,”  and  “we  hope.”  Appendix  B  provides  an  expanded  list  of  narrative 
statements  to  avoid. 

Fourth,  is  the  narrative  for  each  performance  area  of  a  robust  and 
comprehensive  nature,  as  described  in  the  three  examples  of  the  CPAR  Quality 
Checklist,  labeled  “Block  20  -  insufficient  narrative  vs.  a  better  way  to  write  this”? 
Appendix  C  provides  specific  examples  of  insufficient  narratives. 

Fifth,  could  a  contracting  layman  understand  the  work  that  has  been 
performed?  This  will  not  require  an  opening  statement  giving  a  synopsis  of  the 
contract  in  the  narrative  because  that  would  be  part  of  a  separate  data  field  defined 
as  a  sample  contract  effort  description  within  CPARS  (Block  17)  outside  the 
narrative  portion  (Block  20).  Rather,  the  question  would  answer  whether  there  was 
an  excessive  use  of  jargon  or  acronyms  in  the  narrative,  to  the  point  that  even 
someone  familiar  with  common  terms  associated  with  contracting  would  be  unable 
to  understand  the  discussion  in  the  narrative. 

Once  the  yes  or  no  questions  were  scored,  the  following  two  questions  were 
answered  with  a  possible  score  of  1  to  5. 

First,  is  the  narrative  beneficial  to  a  user  of  the  information  above  and  beyond 
the  objective  scores?  To  earn  a  score  of  1 ,  the  narrative  provides  an  unsatisfactory 
amount  of  beneficial  data  to  the  user  above  and  beyond  what  could  be  gleaned  from 
looking  over  the  objective  scores  assigned  in  Block  18.  To  earn  a  score  of  2,  the 
narrative  provides  a  marginal  amount  of  beneficial  data  to  the  user  above  and 
beyond  what  could  be  gleaned  from  looking  over  the  objective  scores  assigned  in 
Block  18.  To  earn  a  score  of  3,  the  narrative  provides  a  satisfactory  amount  of 
beneficial  data  to  the  user  above  and  beyond  what  could  be  gleaned  from  looking 
over  the  objective  scores  assigned  in  Block  18.  To  earn  a  score  of  4,  the  narrative 
provides  a  very  good  amount  of  beneficial  data  to  the  user  above  and  beyond  what 
could  be  gleaned  from  looking  over  the  objective  scores  assigned  in  Block  18.  To 
earn  a  score  of  5,  the  narrative  provides  an  exceptional  amount  of  beneficial  data  to 
the  user  above  and  beyond  what  could  be  gleaned  from  looking  over  the  objective 
scores  assigned  in  Block  18. 
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Second,  does  the  narrative  correlate  to  the  objective  scores  that  have  been 
assigned?  To  earn  a  score  of  1 ,  more  than  one  of  the  performance  areas  described 
in  the  narrative  is  contradictory  to  the  objective  scores  assigned  in  Block  18.  To 
earn  a  score  of  2,  one  (but  no  more  than  one)  of  the  performance  areas  described  in 
the  narrative  is  contradictory  to  the  objective  score  assigned  in  Block  18.  To  earn  a 
score  of  3,  the  narrative  is  merely  satisfactory  in  describing  accurately  why  the 
objective  scores  are  assigned  as  they  are  in  Block  18.  To  earn  a  score  of  4,  the 
narrative  is  very  good  in  describing  accurately  why  the  objective  scores  are  assigned 
as  they  are  in  Block  18.  To  earn  a  score  of  5,  the  narrative  is  exceptional  in 
describing  accurately  why  the  objective  scores  are  assigned  as  they  are  in  Block  18. 

The  next  step  in  conducting  our  analysis  of  the  narratives  was  to  figure  out 
the  best  way  to  handle  the  sheer  volume  of  715  report  cards  while  at  the  same  time 
thoroughly  reviewing  each  report  card.  Software  was  an  initial  thought,  but  after 
looking  at  this  option  in  depth,  it  became  obvious  that  the  process  of  learning  how  to 
use  and  manipulate  the  software  to  give  the  desired  output  would  be  more  time 
consuming  than  it  was  worth.  For  this  sample  size,  it  was  best  to  conduct  an  Inter- 
Rater  Reliability  Test  (Gwet,  2008)  to  determine  a  baseline  agreement  amongst  the 
raters  of  the  questions  outlined  previously,  and  then  go  through  each  narrative 
thoroughly  by  the  raters.  If  this  could  prove  that  the  raters  were  trained  and  could 
rate  each  sample  similarly,  then  each  of  the  715  samples  would  not  need  to  be 
analyzed  by  each  researcher.  Using  this  approach,  the  715  samples  could  be  split 
up  amongst  the  raters  and  would  yield  similar  results  across  each  rater  for  the  same 
sample. 

After  conducting  training  on  the  procedures  outlined  above  in  conducting 
each  score,  a  sample  of  30  reports  was  randomly  selected  from  the  original  715  and 
the  reports  were  independently  rated  by  each  rater.  Appendix  A  provides  the  results 
of  the  Inter-Rater  Reliability  Testing. 

To  score  each  set  of  scores  for  the  yes  or  no  questions,  a  score  of  3  was 
given  if  all  three  of  the  scores  matched.  The  only  other  possibility  for  the  yes  or  no 
questions  is  for  two  of  the  scores  to  match  and  the  other  to  not  match  (highlighted  in 
red).  In  this  case,  a  score  of  1  was  assigned  since  two  of  the  scores  had  a  50%  rate 
of  agreeing  with  the  other  scores  while  the  remaining  score  had  a  0%  rate  of 
agreeing  with  the  other  scores.  In  the  end,  each  set  of  scores  was  summed  and 
then  divided  by  90  to  get  the  total  agreeability  percentage  in  each  category.  The 
acceptable  threshold  was  set  at  whether  all  raters  could  match  scores  for  each 
sample  with  a  minimum  of  80%  reliability. 

The  results  for  the  yes  or  no  questions  show  the  percentage  of  agreeability 
between  each  rater’s  scores  of  the  30  randomly  selected  reports.  There  was  100% 
agreeability  on  whether  the  narrative  addresses  all  performance  areas  assessed. 
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There  was  84.44%  agreeability  on  whether  the  narrative  was  based  on  objective 
data.  There  was  97.877%  agreeability  on  whether  the  narrative  was  free  of 
statements  to  avoid.  There  was  80%  agreeability  on  whether  the  narrative  was 
robust  and  comprehensive.  There  was  91.11%  agreeability  on  whether  a 
contracting  layman  could  understand  the  work  performed. 

For  the  questions  that  were  assigned  scores  of  1  to  5,  the  scores  show  the 
percentage  agreeability  of  whether  the  raters  could  agree  in  their  results  while 
allowing  for  a  1  point  difference  amongst  raters  with  no  penalty.  The  threshold  was 
set  at  whether  all  raters  could  match  scores  for  each  sample  within  a  1  point 
difference  at  a  minimum  80%  reliability.  If  a  set  of  scores  all  matched,  a  score  of  3 
was  given  for  the  set.  If  a  set  of  scores  had  two  occurrences  of  scores  within  1  (for 
example  1 , 2,  3  or  3,  4,  5),  a  score  of  2  was  given  for  the  set.  If  a  set  of  scores  had 
only  one  occurrence  of  scores  within  1  (example  2,  4,  5),  a  score  of  1  was  given  for 
the  set.  If  there  had  been  an  instance  where  scores  of  1 ,  3,  and  5  had  been  given,  a 
score  of  0  would  have  been  given  for  the  set  (this  did  not  occur). 

The  results  show  the  percentage  of  agreeability  between  each  of  the  rater’s 
scores  within  a  score  of  1  based  on  the  30  randomly  selected  samples.  There  was 
93.33%  agreeability  within  1  on  whether  the  narrative  is  beneficial  above  and 
beyond  the  objective  scores.  There  was  80%  agreeability  within  1  on  whether  or  not 
the  narrative  correlates  to  the  objective  scores  assigned. 

Because  all  of  the  total  agreeability  percentages  were  able  to  meet  the 
minimum  80%  threshold,  each  rater  went  through  -230  of  the  remaining  report  cards 
independently  and  the  results  were  combined  and  analyzed  as  if  all  three  raters  had 
gone  through  all  715  report  cards.  The  next  section  discusses  the  methodology  for 
the  statistical  analysis  of  the  database. 

C.  METHODOLOGY  FOR  DATABASE  STATISTICAL  ANALYSIS 

An  independent  2-sample  test  for  proportions  was  performed  to  find  the 
statistical  significance  of  the  proportions  of  the  outcomes  for  successful  versus 
unsuccessful  contracts  for  the  questions  that  have  only  two  possible  outcomes  (yes 
or  no).  The  p  value  in  the  hypothesis  test  shows  the  level  of  statistical  significance.  A 
p  value  of  less  than  .05  shows  statistical  significance.  A  p  value  of  less  than  .01 , 
therefore,  shows  a  much  higher  degree  of  statistical  significance  than  even  the  .05 
values. 

A  chi-square  test  was  performed  to  show  collectively  whether  the  results  for 
the  questions  that  have  five  possible  outcomes  were  statistically  different  for 
successful  versus  unsuccessful  contracts.  The  p  value  used  in  the  chi-square  test 
shows  the  level  of  statistical  significance  and  the  results  are  interpreted  in  the  same 
way  as  in  the  hypothesis  test  that  compares  two  independent  proportions  (explained 
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in  the  previous  paragraph).  The  next  section  discusses  the  interview  questions  that 
were  asked  government  contracting  personnel. 

D.  INTERVIEWS  OF  GOVERNMENT  CONTRACTING  PERSONNEL 

In  support  of  the  data  analysis,  interviews  with  contracting  professionals  from 
two  DoD  contracting  agencies  were  conducted.  These  interviews  consisted  of  a 
series  of  questions  asking  the  subjects  how  they  use  and  to  what  extent  they  value 
CPARS  and  other  sources  of  contractor  past  performance  information  and  to  what 
extent  they  value  the  narratives  compared  to  the  performance  objective  scores. 

Our  research  includes  the  study  of  human  subjects  and  therefore,  as  directed 
by  regulation  from  the  Department  of  Health  and  Human  Services,  the  research 
interviews  required  approval  from  an  institutional  review  board  (IRB)  under  the 
guidance  of  the  NPS  Office  for  Human  Research  Protections  (OHRP)  specialist. 

The  purpose  of  the  IRB  is  to  protect  human  subjects  in  research  and  to  ensure  that 
basic  ethical  principles  are  adhered  to. 

We  conducted  interviews  at  two  geographically  diverse  DoD  contracting 
agencies.  The  interviewees  had  some  combination  of  significant  experience 
submitting  contractor  past  performance  information  to  CPARS  and/or  retrieving  from 
the  Past  Performance  Information  Retrieval  System  (PPIRS)  past  performance 
information  for  use  in  contract  administration  or  source  selection.  Interview  subjects 
were  chosen  by  their  respective  department  heads. 

E.  INTERVIEW  QUESTION  DEVELOPMENT  METHODOLOGY 

Interviews  with  contracting  professionals  who  are  experienced  with  the 
CPARS  and  PPIRS  systems  provided  the  requisite  context  and  depth  of  knowledge 
needed  to  accurately  and  robustly  analyze  and  interpret  the  statistical  data  derived 
from  the  CPARS  narratives.  The  specific  interview  questions  were  designed  to 
avoid,  to  the  extent  practicable,  injection  of  bias  by  steering  respondents. 

Interviewing  subjects  at  two  government  agencies  with  significantly  different 
missions  and  contract  award  types  facilitated  the  collecting  of  information  across  a 
wide  spectrum  of  knowledge  of  the  CPARS/PPIRS  systems.  Interviewing  individuals 
with  varying  roles  in  collecting,  submitting,  and  utilizing  contractor  past  performance 
information  provided  us  with  responses  across  a  diverse  spectrum  of  experience.  A 
complete  listing  and  explanation  of  the  interview  questions  can  be  found  in  Appendix 
D. 


Following  this  second  phase  of  research,  the  study  moved  into  analyzing 
CPARS  contractor  past  performance  report  card  narratives  and  comparing  that  data 
to  the  information  retrieved  from  these  contracting  professional  interviews. 
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F.  SUMMARY 

This  chapter  presented  an  overview  of  the  methodology  for  collecting  the 
data,  analyzing  the  CPARS  narratives,  and  conducting  interviews  of  DoD  agency 
contracting  professionals.  The  objective  scores  were  introduced  and  discussed. 
Next,  the  chapter  reviewed  the  CPAR  Quality  Checklist  to  provide  a  framework  for 
analyzing  the  database  which  is  the  basis  for  this  research  project.  The 
methodology  for  the  database  statistical  analysis  was  then  explained.  Finally,  the 
chapter  concluded  with  a  review  of  interview  questions  posed  to  DoD  contracting 
professionals.  The  next  chapter  discusses  the  results  of  the  statistical  analysis,  the 
responses  to  interview  questions  by  DoD  contracting  professionals,  and  the  findings 
that  answer  the  research  questions  posed  in  Chapter  III. 
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IV.  ANALYSIS 


A.  INTRODUCTION 

This  chapter  reveals  the  results  of  the  analysis  that  examined  the  value  of 
CPARS  report  card  narratives  for  service  contracts  as  they  relate  to  their  associated 
objective  scores.  This  chapter  begins  with  an  overview  of  the  data  used  to  support 
the  research  presented  in  this  report.  The  overview  first  describes  the  database 
created  for  this  report  in  order  to  show  from  where  the  results  of  the  statistical 
analysis  were  derived.  The  overview  then  provides  a  description  of  the  series  of 
interviews  that  were  conducted  to  show  from  where  the  results  of  the  interview 
findings  came.  Once  the  database  and  the  interviews  are  described,  the  chapter 
explains  the  statistical  analysis  findings  in  detail.  Following  the  description  of  the 
statistical  analysis  findings,  this  chapter  describes  the  interview  findings  with  each 
finding  supported  by  specific  commentary  from  the  interviews. 

B.  OVERVIEW  OF  THE  DATA 

The  database  used  for  the  statistical  analysis  portion  of  this  project  is  limited 
to  the  715  Army  service  contracts  analyzed  by  a  previous  NPS  MBA  project  (Hart  et 
al. ,  2013).  The  selection  criteria  for  those  715  contracts  was  limited  to  Army  service 
contracts,  “contracts  only  submitted  by  Mission  Installation  Contracting  Commands 
(MICCs),  and  contracts  from  only  five  MICC  offices”  (Hart  et  al.,  2013).  Of  note,  out 
of  the  715  contract  reports  contained  in  the  database,  “22  resulted  in  failure  resulting 
in  a  total  contract  failure  rate  of  3.08%”  (Hart  et  al.,  2013).  The  determination  for 
which  contracts  should  be  specified  as  failures  “was  determined  by  whether  a 
contract  received  a  marginal  or  unsatisfactory  objective  rating  in  quality  of 
product/service,  schedule,  cost  control,  business  relations,  management  of  key 
personnel,  or  utilization  of  small  business”  (Hartetal.,  2013). 

Interviews  with  contracting  professionals  from  two  DoD  contracting  agencies 
were  conducted  and  provided  the  basis  for  the  interview  analysis.  These  interviews 
consisted  of  a  series  of  questions  asking  the  subjects  how  they  use  and  to  what 
extent  they  value  CPARS  and  other  sources  of  contractor  past  performance 
information  and  to  what  extent  they  value  the  narratives  compared  to  the 
performance  objective  scores.  Appendix  D  provides  the  specific  questions  asked 
during  the  interviews  and  the  reasoning  behind  those  questions. 

C.  STATISTICAL  ANALYSIS  FINDINGS 

Next,  we  reveal  the  findings  from  our  statistical  analysis.  First,  we  describe 
each  of  those  findings  in  detail.  We  cover  whether  the  narratives  address  all 
performance  areas  assessed,  whether  the  narratives  are  based  on  objective  data, 
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whether  the  narratives  are  free  of  statements  to  avoid,  whether  the  narratives  are 
robust  and  comprehensive,  and  whether  the  narratives  are  written  so  that  a 
contracting  layman  should  understand  the  work  performed.  Then,  we  cover  whether 
the  narrative  provides  beneficial  data  above  and  beyond  what  could  be  gleaned  from 
looking  over  the  objective  scores  assigned.  Lastly,  we  cover  how  well  the  narrative 
sentiment  matches  up  with  the  objective  scores  assigned.  At  the  end  of  this  section, 
a  summary  chart  of  the  statistical  analysis  findings  is  provided  in  Table  7. 

1.  Narrative  Addresses  All  Performance  Areas  Assessed? 

Overall,  the  narratives  addressed  all  performance  areas  assessed  82%  of  the 
time.  This  means  that  -18%  of  the  time,  contracting  professionals  are  not  submitting 
to  CPARS  contractor  performance  narratives  in  accordance  with  the  guidelines 
provided  in  the  CPARS  user’s  manual  for  this  requirement.  This  was  less 
problematic  with  unsuccessful  contracts  at  -95%  than  with  successful  contracts  at 
-81%.  The  difference  in  the  proportion  of  times  that  the  narrative  addressed  all 
performance  areas  assessed  in  successful  and  unsuccessful  contracts  is  statistically 
significant  (p  <  .05). 

2.  Narrative  Is  Based  on  Objective  Data? 

Overall,  the  narratives  were  based  on  objective  data  -77%  of  the  time.  This 
means  that  -23%  of  the  time,  contracting  professionals  were  not  submitting 
toCPARS  contractor  performance  narratives  in  accordance  with  the  guidelines 
provided  in  the  CPARS  user’s  manual  for  this  requirement.  However,  in 
unsuccessful  contracts,  the  narratives  were  based  on  objective  data  1 00%  of  the 
time.  This  is  significantly  different  from  the  -77%  in  successful  contracts  (p  <  .01 ). 

3.  Narrative  Is  Free  of  Statements  to  Avoid? 

Overall,  the  narratives  were  free  of  statements  to  avoid  -97%  of  the  time. 

This  means  that  -3%  of  the  time,  contracting  professionals  were  not  submitting  to 
CPARS  contractor  performance  narratives  in  accordance  with  the  guidelines 
provided  in  the  CPARS  user’s  manual  for  this  requirement.  This  was  slightly  more 
problematic  with  unsuccessful  contracts  at  -86%  than  with  successful  contracts  at 
-97%  (p<  .01). 

4.  Narrative  Is  Robust  and  Comprehensive? 

Overall,  the  narratives  were  robust  and  comprehensive  -63%  of  the  time. 

This  means  that  -37%  of  the  time,  contracting  professionals  were  not  submitting  to 
CPARS  contractor  performance  narratives  in  accordance  with  the  guidelines 
provided  in  the  CPARS  user’s  manual  for  this  requirement.  This  was  less 
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problematic  with  unsuccessful  contracts  at  -91%  than  with  successful  contracts  at 
-62%  (p  <  -01). 

5.  Could  a  Layman  Understand  Work  Performed? 

Overall,  the  narratives  were  written  so  that  a  contracting  layman  should 
understand  the  work  performed  -64%  of  the  time.  This  means  that  -36%  of  the  time, 
contracting  professionals  were  not  submitting  to  CPARS  contractor  performance 
narratives  in  accordance  with  the  guidelines  provided  in  the  CPARS  user’s  manual 
for  this  requirement.  This  was  less  problematic  with  unsuccessful  contracts  at  -82% 
than  it  was  with  successful  contracts  at  -64%  (p  <  .05). 

6.  Is  the  Narrative  Beneficial  Above  and  Beyond  Objective  Scores? 

As  previously  described  in  the  methodology  chapter,  the  chi-square  test 
determines  whether  distributions  of  scores  are  statistically  different.  Using  this  test, 
we  determined  that  there  was  a  difference  between  successful  and  unsuccessful 
contracts  in  whether  the  narratives  were  beneficial  above  and  beyond  the  objective 
scores.  Unsuccessful  contracts  tended  to  have  more  beneficial  CPAR  report  card 
narratives  than  successful  contracts. 

Overall,  the  narrative  provided  an  unsatisfactory  amount  of  beneficial  data  to 
the  user  -12%  of  the  time.  However,  there  were  no  unsuccessful  contracts  that 
provided  an  unsatisfactory  amount  of  beneficial  data.  The  narrative  provided  a 
marginal  amount  of  beneficial  data  -22%  of  the  time.  There  were  no  unsuccessful 
contracts  that  provided  a  marginal  amount  of  beneficial  data.  The  narrative  provided 
a  satisfactory  amount  of  beneficial  data  -28%  of  the  time.  The  narrative  provided  a 
very  good  amount  of  beneficial  data  -21%  of  the  time.  The  narrative  provided  an 
exceptional  amount  of  beneficial  data  -18%  of  the  time.  This  was  much  more  likely 
to  occur  with  unsuccessful  contracts  than  with  successful  contracts  at  -17%. 

7.  Does  the  Narrative  Correlate  to  Objective  Scores  Assigned? 

We  again  used  the  chi-square  test  and  determined  that  there  was  not  a 
difference  between  successful  and  unsuccessful  contracts  in  whether  the  narrative 
correlates  to  the  objective  scores  assigned. 

Overall,  the  narrative  sentiment  was  contradictory  to  more  than  one  of  the 
objective  scores  assigned  -2%  of  the  time.  The  narrative  sentiment  was 
contradictory  to  one  of  the  objective  scores  assigned  -6%  of  the  time.  The  narrative 
sentiment  was  satisfactory  in  describing  accurately  why  the  objective  scores  were 
assigned  as  they  were  -28%  of  the  time.  The  narrative  sentiment  was  very 
successful  in  describing  accurately  why  the  objective  scores  were  assigned  as  they 
were  -40%  of  the  time.  The  narrative  sentiment  was  exceptionally  successful  in 
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describing  accurately  why  the  objective  scores  were  assigned  as  they  were  -24%  of 
the  time. 

Table  7  is  a  summary  chart  of  the  statistical  database  analysis  that  we  have 
just  described  in  detail. 

Table  7.  Results  of  Statistical  Database  Analysis 


Result 

Overall 

Unsuccessful 

Contract 

Successful 

Contract 

Successful  vs 

Unsuccessful 

Statistically 

Significant? 

P  Value 

Narrative 

Addresses  All 

Performance 

Areas  Assessed? 

No 

18.46% 

4.55% 

18.90% 

Yes 

0.043 

Yes 

81.54% 

95.45% 

81.10% 

Narrative  Based 

On  Objective 

Data? 

No 

22.52% 

0.00% 

23.23% 

Yes 

<.01 

Yes 

77.48% 

100.00% 

76.77% 

Narrative  Is  Free 

Of  Statements  To 

Avoid? 

No 

3.08% 

13.64% 

2.74% 

Yes 

<.01 

Yes 

96.92% 

86.36% 

97.26% 

Narrative  Is 

Robust  & 
Comprehensive? 

No 

37.48% 

9.09% 

38.38% 

Yes 

<.01 

Yes 

62.52% 

90.91% 

61.62% 

Could  A  Layman 

Understand  Work 

Performed? 

No 

35.52% 

18.18% 

36.08% 

Yes 

0.042 

Yes 

64.48% 

81.82% 

63.92% 

Is  The  Narrative 

Beneficial  Above 
&  Beyond 
Objective  Scores? 

Score  1 

11.61% 

0.00% 

12.12% 

Yes 

<.01 

Score  2 

21.68% 

0.00% 

22.37% 

Score  3 

27.97% 

18.18% 

28.28% 

Score  4 

20.84% 

27.27% 

20.63% 

Score  5 

17.76% 

54.55% 

16.59% 

Does  The 

Narrative 

Correlate  To 

Objective  Scores 
Assigned? 

Score  1 

1.82% 

4.55% 

1.88% 

No 

0.141 

Score  2 

6.43% 

4.55% 

6.49% 

Score  3 

27.83% 

9.09% 

28.43% 

Score  4 

40.42% 

40.91% 

40.40% 

Score  5 

23.36% 

40.91% 

22.80% 
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D. 


INTERVIEW  FINDINGS 


As  previously  discussed,  we  interviewed  contracting  professionals  from  two 
geographically  diverse  DoD  contracting  agencies.  We  did  this  to  find  out  how  they 
use  and  to  what  extent  they  value  CPARS  and  other  sources  of  contractor  past 
performance  information.  We  also  wanted  to  find  out  to  what  extent  they  value  the 
narratives  compared  to  the  objective  scores.  In  the  process  of  conducting  these 
interviews,  we  made  some  interesting  discoveries  that  could  be  useful  in  improving 
the  contracting  process  specifically  related  to  the  narrative  portion  of  past 
performance  assessment  reports.  Below  are  the  eight  findings  along  with  specific 
excerpts  from  the  interviews  supporting  each  finding. 

1.  CPARS  Is  Still  Often  Not  Reliable,  Robust,  or  Comprehensive 
Enough. 

The  information  in  CPARS/PPIRS  is  oftentimes  not  reliable,  robust,  or 
comprehensive  enough  to  allow  source  selection  officials  to  place  a  significant 
enough  weight  on  past  performance.  This  finding  was  also  noted  in  several  GAO 
and  OFPP  memorandums  mentioned  earlier  in  the  literature  review  chapter  of  this 
report. 

The  Past  Performance  Information  Retrieval  System  (PPIRS)  is  an 
information  system  that  draws  from  the  reports  entered  into  CPARS.  The  contracting 
professionals  we  interviewed  who  deal  regularly  with  utilizing  the  data  in  PPIRS  felt 
that  if  the  information  in  PPIRS  were  more  reliable  and  consistent,  they  would  be 
able  to  give  more  weight  to  it  when  making  source  selection  decisions.  The 
interviewees  stated  that  when  they  knew  there  should  be  something  in  PPIRS  on  a 
particular  contractor  for  a  particular  type  of  contract  and  dollar  amount,  they  would 
often  search  for  the  contractor’s  name  in  CPARS,  only  to  receive  this  response,  “no 
information  has  been  entered.” 

Interviewee  Number  2  made  the  recommendation  that  PPIRS  would  be  a 
more  useful  tool  if  source  selection  teams  could  get  more  information  in  there  from 
end  users.  Interviewee  Number  2  was  hesitant  because  this  recommended 
improvement  will  likely  create  more  work  for  both  the  end  user  and  the  contract 
specialist,  but  suggested  that  the  DoD  may  want  to  lower  the  dollar  value  threshold 
to  help  contracting  offices  that  normally  deal  with  lower  value  contracts. 

Interviewee  Number  5  stated  that  the  CORs  and  contracting  officers  know 
generally  how  the  report  is  supposed  to  be  done,  “but  it  is  just  something  that  is  not 
executed  as  well  as  it  could  be.”  Interviewee  Number  5  stated  that  information  from 
CPARS/PPIRS  is  not  really  helpful  in  source  selection  and  contract  administration 
because  the  database  is  not  maintained  well  enough  and  not  enough  thought  is  put 
into  them.  Interviewee  Number  5  stated, 
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Mostly  what  I  see  is  there  is  either  satisfactory  or  excellent  reviews 
even  if  there  are  problems,  the  problems  are  not  always  documented 
and  I  think  that  in  one  case  at  least  it  is  actually  the  same  CPARS 
pasted  into  each  year,  each  period  and  you  could  tell  that  even  the 
dates  were  actually  off  sometimes.  So  it  is  not  something  that  we  put 
enough  thought  into,  which  makes  it  not  a  very  useful  tool  in  source 
selections,  especially  when  you  know  other  government  agencies 
might  be  a  little  bit  similar  in  their  practices  and  it  is  not  really 
something  ...  and  it  is  something  that  you  need  to  take  with  a  grain  of 
salt.  I  think  when  we  are  going  through  source  selection  really  what 
we  look  for  is  the  offeror’s  technical  understanding  of  the  requirements. 
Sometimes  it  is  obvious  that  there  is  a  little  bit  more  business 
development  language  than  actual  understanding  of  the  work  and  that 
seems  to  be  more  of  the  focus  than  the  actual  CPARS. 

We  also  noted  similar  issues  in  conducting  our  database  analysis,  that  in  some 
cases  the  exact  or  almost  the  exact  same  narrative  was  cut  and  pasted  into  several 
different  contract  report  cards. 

2.  Unsuccessful  Contracts  Are  More  Reliable,  Robust,  and 
Comprehensive. 

Unsuccessful  contracts  tend  to  have  more  reliable,  robust,  and 
comprehensive  past  performance  information  available  in  their  CPARS/PPIRS 
reports. 

Interviewee  Number  3  stated  that  there  might  be  a  reason  why  the 
unsuccessful  contracts  were  typically  more  robust  and  comprehensive  than  the 
successful  ones: 

If  the  contractor  is  really  just  coasting  along  doing  everything  really, 
really  well,  I  think  a  lot  of  times  you  might  not  have  a  really  in-depth 
report  because  the  COR  says,  “Hey,  everything’s  good.  I  don’t  have  to 
worry  about  anything.”  So,  they’re  just  going  to  kind  of  mark  it  up, 

“Hey,  they’re  good  or  they’re  bad  or  whatever  on  everything.”  I  think 
the  ones  that  maybe  would  take  longer  is  when  you  have  issues. 

Interviewee  Number  3  went  on  further  to  state  that  narratives  might  be  longer  for  a 
poor  performance  because 

they  know  that  those  are  going  to  get  ...  the  contractor  is  going  to  be 
able  to  review  those  as  well  as  the  contracting  officer,  so  the  COR 
should  put  more  time  into  those  to  just,  you  know,  really  explain 
himself  why  you’re  giving  a  poor  rating.  Typically,  if  you’re  getting  a 
good  rating,  the  contractor  is  just  going  to  say,  “Yup,  I  agree.  Go 
ahead  and  post  it  to  PPIRS.  I  agree  with  it.”  But  if  you’re  giving  them  a 
negative  rating,  you’re  really  going  to  want  to  explain  yourself. 
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3.  Assigning  Weight  to  Past  Performance  Using  the  Information 
Available 

The  appropriate  amount  of  weight  that  should  be  assigned  to  past 
performance  in  making  source  selection  decisions  should  be  correlated  to  the 
source,  availability,  quality,  and  relevancy  of  the  past  performance  information. 

Interviewee  Number  3  made  the  comment  that  the  weight  assigned  to 
contractor  past  performance  in  PPIRS  is  basically  a  direct  correlation  to  the 
relevancy  of  the  past  performance  combined  with  how  robust  and  comprehensive 
the  written  narrative  is. 

Interviewee  Number  5  stated  that  if  they  had  negative  past  performance 
information  from  PPIRS  or  from  a  questionnaire,  they  would  give  it  more  weight,  but 
generally  they  have  to  give  more  weight  to  the  technical  portions  of  a  proposal 
because  PPIRS  isn’t  able  to  produce  the  information  they  need  and  the 
questionnaire  is  not  likely  to  produce  negative  past  performance  information. 

Interviewee  Number  6  stated  that  it  is  more  beneficial  if  the  source  selection 
authority  is  able  to  access  multiple  contracts  of  similar  scope  and  complexity. 
Interviewee  Number  6  stated,  “If  there  is  consistency  I  would  place  greater  value  in 
that  evaluation.  If  they  are  all  over  the  place  then  it  is  really  hard  to  put  a  ...  it  is  kind 
of  subjective  how  much  value  you  put  into  a  particular  CPAR.” 

4.  What  to  Do  With  Contradictory  Past  Performance  Information 

PPIRS  sometimes  contains  information  in  the  narrative  that  is  either 
contradictory  or  does  not  quite  match  up  with  the  objective  scores.  When  the 
objective  scores  and  narrative  sentiment  in  PPIRS  are  mismatched,  contracting 
professionals  tend  to  give  more  weight  to  the  narrative  versus  the  objective  scores. 

Interviewee  Number  1  made  the  strong  recommendation  that  the  contracting 
professionals  who  regularly  deal  with  PPIRS  in  looking  for  past  performance 
information  need  to  take  note  of  when  there  is  conflicting  information  between  the 
objective  scores  assigned  and  the  written  narrative.  That  interviewee  said  that  when 
source  selection  team  members  find  something  like  that,  they  should  send  the  report 
back  to  whomever  entered  the  information  and  make  them  correct  that  section.  This 
comment  was  reiterated  by  Interviewee  Number  4: 

I’d  send  it  back  and  tell  whoever  wrote  it  to  come  up  with  why  their 
score  was  what  it  is  and  explain  to  them,  look,  I’m  looking  at  this  and 
I’m  reviewing  what  you  wrote,  but  it  doesn’t  justify  the  outstanding 
rating  or  the  poor  rating. 

Interviewee  Number  2  stated  that  they  give  considerably  more  weight  to  the 
narrative  versus  the  objective  scores  because 
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sometimes  you  will  see  the  government  rated  them  very  good,  but  then 
the  description  you’re  reading  is  like  ...  it  doesn’t  match  ...  there’s  a 
description  next  to  what  very  good  should  be  and  then  what  they’re 
saying  doesn’t  match  very  good. 

Interviewee  Number  2  also  stated  that  the  narrative  is  usually  more  beneficial 
because  sometimes 

there’s  a  bunch  of  different  categories  and  depending  on  what  the 
contract  is  for,  a  lot  of  them  may  be  N/A,  so  the  description  is  a  lot 
better  than  just  saying  like  ...  like  if  you’re  doing  a  firm  fixed  price 
contract  but  for  some  reason  they  rate  cost  control  and  they  say, 
“excellent,”  well,  I  would  hope  so  because  we’re  only  paying  the  firm 
fixed  price. 

5.  Reliable  and  Accurate  Past  Performance  Retrieval  Methods  Are 
Needed. 

Contractor  past  performance  reports  might  not  be  as  accessible  as  they 
should  be  in  PPIRS  because  contracting  professionals  are  not  always  applying  due 
diligence  in  identifying  the  appropriate  entity  using  identifying  information  within  the 
contractor’s  parent  organization.  Examples  of  such  identifying  information  would  be 
an  appropriate  cage  code  or  DUNS  (Dun  and  Bradstreet  Data  Universal  Numbering 
System)  number. 

Interviewee  Number  3  alluded  to  the  fact  that  it  might  be  more  beneficial  to 
have  the  PPIRS  database  set  up  to  more  easily  find  similar  contracts  for  a  given 
contractor.  Interviewee  Number  3  stated  that 

one  of  the  things  we’re  really  looking  for  when  we’re  doing  our  source 
selection  is  the  relevancy  of  past  performance,  so  not  just,  “Hey,  they 
performed  well  on  a  bunch  of  contracts.”  That  matters,  but  not  as 
much  as  it  would  matter  if  they  performed  really  well  on  a  similar  type 
contract  to  the  one  that  we’re  looking  to  award.  So,  you  know,  you 
may  have  a  contract  for  a  completely  different  service  that  this 
company  did  and  they  got  really  awesome  ratings,  that  gives  us  an 
idea  that  they’re  able  to  perform  well  and  are  probably  good  workers, 
but  what  does  that  really  tell  us  about  how  they’re  going  to  perform  this 
contract  that  is  a  completely  different  type  of  service.  So,  we  use  it, 
but  I  would  say  we’re  going  to  ...  we  have  other  methods  of  doing  past 
performance  research  to  look  at  their  relevancy  and,  you  know  ... 
yeah,  their  relevancy  to  the  contract  that  we’re  looking  to  award.  To 
me  that’s  more  important.  It’s  how  will  they  perform  on  a  similar  type 
contract.  But  if  they’ve  never  done  this  type  of  work  before,  yeah,  then 
it  would  be  weighed  more  heavily  than  this  stuff  that  you’re  looking  at 
in  PPIRS  because  that’s  all  you  really  have  to  go  on.  So,  I  think  the 
reports  are  good  enough  to  explain,  hey,  this  is  the  type  of  work  that’s 
being  done,  so  you  can  really  make  a  good  comparison. 
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We  noted  that  while  this  might  give  the  contracting  professionals  information 
to  use  in  weighing  past  performance,  the  information  provided  as  references  from 
contractors  would  almost  certainly  be  skewed  to  positive  references.  A  contractor 
will  most  likely  omit  a  recent  contract  reference  that  they  know  they  performed  poorly 
on.  This  is  really  where  the  process  of  incorporating  past  performance  information 
into  source  selection  is  failing  in  that  PPIRS  should  be  the  avenue  for  source 
selection  officials  to  find  negative  past  performance  information  if  it  exists.  Too  often, 
it  looks  like  data  in  the  system  are  not  organized  well  enough.  The  way  the  data  are 
organized  is  not  helping  source  selection  officials  mine  the  database  to  uncover  this 
recent  and  relevant  negative  past  performance  information  as  well  as  it  should. 

When  asked  for  ideas  of  how  CPARS/PPIRS  could  be  improved,  Interviewee 
Number  9  had  the  recommendation  that  “there  could  be  something  in  the  PPIRS 
system  that  would  allow  you  ...  to  weed  through  the  relevancy  aspect.” 

6.  Past  Performance  Questionnaires  Are  inadequate. 

Source  selection  officials  often  need  to  solicit  contractors  for  references  or 
ask  them  to  fill  out  a  past  performance  questionnaire  because  there  is  a  lack  of 
reliable,  robust,  and  comprehensive  past  performance  information  available  in 
PPIRS. 

Because  they  are  often  unable  to  find  relevant,  robust,  and  comprehensive 
information  about  a  particular  proposal  in  PPIRS,  Interviewee  Number  3  stated, 

In  our  solicitations  or  our  RFPs,  when  we  put  those  out  then  we’ll  ask 
for  the  contractor  to  submit  ...  for  example,  we  might  ask  them  to 
submit  five  of  their  most  recent  contracts  that  they’ve  done  for  similar 
type  work  with  references.  So,  they  would  provide  references  of,  you 
know,  John  Doe  who  they  did  work  for  in,  you  know,  for  NAVSEA  out 
of  Washington,  DC,  or  something  like  that.  I  could  contact  him  and 
say,  “Hey,  how  did  these  guys  perform  on  this?”  Or,  I  could  send  them 
an  actual  ...  like  a  survey  to  fill  out  and  then  we’ll  get  that  all  filled  out 
and  then  we  know  that  it’s  a  relevant  contract,  we  know  it’s  a  recent 
contract  that  they’ve  done,  So,  it’s  a  lot  easier  to  use  that  information 
than  it  is  a  lot  of  times  in  PPIRS,  because  if  you  go  into  PPIRS  it’s  like 
the  most  recent  one  might  be  like  2009  or  2010  and  it’s  like  a 
completely  different  contract. 

Interviewee  Number  5  made  note  of  a  similar  questionnaire  and  reiterated  the 
issue  with  it  by  stating, 

When  we  have  source  selections  here  at  our  agency,  we  usually  have 
the  contractors  fill  out  a  past  performance  questionnaire.  It  is  a 
standard  questionnaire  that  we  send  out  for  each  offeror  and  a 
government  representative  has  to  fill  them  out  and  send  them  back 
directly  to  the  contracting  officer.  Now  those  are  a  little  bit  different  in 
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that  it  is  not  CPARS,  but  they  do  have  to  reference  it  and  we  do 
validate  that  it  is  accurate.  But  I  mean  I  think  that  sometimes  you  find 
that  of  course  they  are  only  going  to  give  references  for  their  best 
clients  so  they  are  always  going  to  be  good  or  at  least  satisfactory. 

7.  Timely  Reports  Must  Be  of  High  Quality  to  Be  Useful. 

There  has  been  a  recent  increase  in  senior-level  interest  to  complete  required 
CPARS  reports  in  a  timely  manner  but  not  an  increased  interest  in  the  quality  of 
those  reports. 

Interviewee  Number  2  stated  that  the  narratives  are 

not  as  lengthy  as  they  could  be.  Like,  if  you  figure  we  spend  three 
months  on  something,  you  would  think  that  they  would  have  like  a  real 
good  paragraph  ...  very  detailed — but  sometimes  it  seems  like  I  get 
more  information  talking  to  people  than  the  two  sentences  they  put  in 
CPARS. 

Interviewee  Number  7  brought  up  a  noticeably  increased  emphasis  on  timely 
completion  of  the  report  card  but  said  it  was  just  as  important  to  have 

quality,  because  really  you  are  wasting  your  time  if  you  are  just  putting 
this  ...  you  are  wasting  everybody’s  time.  The  emphasis  is,  right  now 
more  timeliness.  We  have  gotten  it  from  very  high  up  that  they  want 
these  done;  they  haven’t  really  said  done  well.  So  I  think  after  we  get 
our  hands  around  and  get  people  doing  it  within  the  timeframe  they  are 
supposed  to  be  doing  it,  then  we  would  like  to  turn  to  the  quality  of 
them.  I  know  some  of  the  field  activities  have  already  taken  that  step. 

I  heard  one  field  activity  actually  had  ...  the  commanding  officer  went 
through  every  single  CPAR  done  in  the  last  year  and  pulled  some  of 
them  back  and  said  that  he  wanted  it  redone  because  it  didn’t  have  a 
narrative  that  supported  the  ratings  they  gave  them.  So  with  that,  if 
you  are  getting  good,  quality  CPARS,  I  think  it  is  a  very  valuable  tool 
and  one  that  should  be  used  in  source  selections  because  it  is  like 
having — it  is  like  the  Angie’s  List  for  the  federal  government  that  you 
have  something  that  you  have  feedback  on  customers. 

8.  Miscellaneous  Recommendations  for  Improvement 

Several  of  the  interviewees  made  recommendations  for  improving  services 
acquisition  related  to  contractor  past  performance  information.  The 
recommendations  were  to  add  more  analysis  tools  and  performance  metrics  in 
PPIRS,  better  monitor  the  workload  of  CORs,  better  train  the  workforce  on  writing 
CPARS  narratives,  and  better  broadcast  the  findings  of  CPARS  Program  Office 
audits. 


Interviewee  Number  6  recommended  that  the  program  office  could 
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load  CPARS  up.  I  mean,  you  could  have  cost  analyses,  you  could 
have  trend  analyses,  and  you  could  have  performance  analyses.  I 
mean  did  they  perform,  how  well  did  they  perform,  did  they  make  their 
timelines,  was  their  product  accepted,  did  it  go  through  developmental 
tests,  operational  tests,  was  it  kicked  back? 

Interviewee  Number  7  mentioned  the  workload  of  the  CORs  as  a  potential 
factor  in  the  inadequacy  of  the  contract  report  cards,  stating, 

I  did  statistics  on  how  many  contracts  and  we  had  some  CORs  that 
were  definitely  way  overloaded.  Based  on  what  they  were  saying, 
really  to  do  a  good  job  on  a  good  contract  you  should  only  be  a  COR 
on  like  one  major  contract.  Smaller  ones,  two  to  three,  but  we  have 
people — six  or  eight,  more  than  that.  So  our  workload  was — I  don’t 
think  that  was  the  reason  they  weren’t  doing  their  CPARS,  but  it  was 
one  of  the  reasons  they  weren’t  doing  all  of  the  things  they  needed  to 
do. 

When  asked  about  ways  that  would  help  improve  CPARS  so  that  it  could  be 
more  useful,  Interviewee  Number  8  stated,  “Training  is  probably  the  biggest  one.” 
Interviewee  Number  8  also  stated,  “We  don’t  train  our  people  very  well  in  writing, 
especially  when  you  have  engineers  that  are  filling  out  some  of  these  things.  They 
can  be  rather  cryptic.” 

Interviewee  Number  8  also  made  the  recommendation  that  when  there  are 
audits  conducted  at  a  command  regarding  CPARS,  it  needs  to  be  better  broadcast 
to  the  people  involved  in  services  acquisition  to  better  gain  access  to  the  results  of 
those  audits  along  with  the  trends  across  the  whole  of  government.  Specifically, 
Interviewee  Number  8  stated, 

the  CPARS  Program  Office  goes  out  and  does  audits  periodically. 
There  really  hasn’t  been — one  time  they  did  a  lessons  learned  and 
publicized  that,  but  it  would  be  good  to  see  some  trends,  things  that 
they  could  suggest  that  they  are  seeing  people  having  a  hard  time 
doing.  You  know,  that  would  help  improve  CPARS  as  a  role  if  they  are 
finding  some  people;  certain  contracts  are  harder  to  do.  I  don’t  know. 
Whatever  they  are  finding  in  their  audits,  we  never  get  the  result. 

E.  SUMMARY 

This  chapter  revealed  the  results  of  the  research  that  examines  the  value  of 
CPARS  report  card  narratives  for  service  contracts  as  they  relate  to  their  associated 
objective  scores.  This  chapter  began  with  an  overview  of  the  data  used  to  support 
the  research  presented  in  this  report.  The  overview  first  described  the  database 
created  for  this  report  in  order  to  show  from  where  the  results  of  the  statistical 
analysis  originated.  The  overview  then  provided  a  description  of  the  series  of 
interviews  that  were  conducted  to  show  from  where  the  results  of  the  interview 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-45- 


findings  came.  Once  the  database  and  the  interviews  were  described,  the  chapter 
explained  the  statistical  analysis  findings  in  detail.  Following  the  statistical  analysis 
findings,  this  chapter  described  the  interview  findings,  with  each  finding  supported  by 
specific  commentary  from  the  interviews.  The  next  chapter  includes  a  summary  of 
the  research,  conclusions  from  the  findings,  and  areas  for  further  research  to 
enhance  the  knowledge  of  service  contracting. 
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V.  SUMMARY,  CONCLUSION,  AND  AREAS  FOR  FURTHER 

RESEARCH 


A.  SUMMARY 

The  DoD  obligated  approximately  $360  billion  in  fiscal  year  2012  for  contracts 
for  supplies  and  services  (GAO,  2013).  In  order  to  support  best  practices  for 
government  acquisition,  the  Office  of  Federal  Procurement  Policy  (OFPP)  directs  in 
FAR  Part  42  that  federal  contracting  agencies  shall  use  CPARS  as  their  primary  tool 
for  documenting  contractor  past  performance. 

CPARS  is  a  system  that  was  designed  to  be  used  by  contracting 
professionals  for  submitting  and  retrieving  contractor  past  performance  information. 
There  have  been  many  discrepancies  and  failures  specified  in  recent  GAO  reports 
with  the  utilization  of  CPARS.  Government  contracting  agencies  are  failing  to 
properly  document  contractor  past  performance  information  within  the  schedule 
requirements  mandated  by  OFPP.  In  a  2009  report,  the  GAO  analyzed  data  from 
2007  and  showed  that  DoD  contracting  components  completed  required  contractor 
past  performance  report  cards  less  than  half  of  the  time  (GAO,  2009). 

Subsequently,  the  DoD  increased  its  focus  on  training  and  education  for 
contracting  professionals,  which  has  led  to  an  increase  in  contractor  performance 
assessments  being  completed  and  submitted  to  PPIRS.  In  the  last  published  report 
during  the  research  period  of  this  project,  the  GAO  (2013)  noted  significant  gains  in 
completion  rates.  Fifty-six  percent  of  required  reports  were  completed  in  201 1  while 
74%  were  completed  in  2013.  However,  according  to  the  same  report,  over  half  of 
these  reports  were  submitted  late.  In  addition,  many  CPARS  report  cards  contain 
narratives  that  are  either  insufficiently  detailed  or  conflict  with  their  associated 
objective  scores.  Late  reports  lacking  sufficient  accurate  information  provide  less- 
than-optimal  information  to  the  contracting  professionals  who  rely  on  these  report 
cards  for  source  selection  and  contract  administration  purposes. 

B.  CONCLUSION 

1.  Research  Findings 

The  purpose  of  this  research  was  to  determine  the  value  of  contractor 
performance  assessment  report  narratives  for  services  contracts  by  comparing  the 
relationships  between  narratives  and  objective  scores  in  order  to  recommend 
improvements  to  the  CPARS  contractor  performance  information  documentation 
process.  To  identify  this  value,  the  research  focused  on  answering  five  research 
questions.  Our  research  questions  and  findings  are  as  follows. 
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•  To  what  degree  are  government  contracting  professionals  submitting 
contractor  performance  narratives  to  CPARS  in  accordance  with  the 
guidelines  provided  in  the  CPARS  user’s  manual? 

To  answer  the  first  research  question,  the  data  indicated  that,  overall, 
contracting  professionals  inputting  reports  into  CPARS  are  doing  a  highly  effective 
job  at  ensuring  that  their  narratives  are  free  of  statements  to  avoid  (-96.9%).  These 
professionals  are  not  doing  as  well  in  addressing  all  performance  areas  assessed  in 
their  objective  scores  (-81 .5%)  or  in  ensuring  that  the  narratives  are  based  on 
objective  data  (-77.5%).  The  areas  that  contracting  professionals  seem  to  have  the 
most  issues  with,  however,  are  writing  a  comprehensive  narrative  (-62.5%)  and 
ensuring  that  a  contracting  layman  (such  as  someone  who  might  need  to  access 
CPARS  data  in  order  to  make  a  decision  about  a  future  contract  award)  can  fully 
understand  the  work  performed  (-64.5%).  With  the  exception  of  the  narratives  being 
free  of  statements  to  avoid,  contracting  professionals  were  more  effective  in  all 
categories  when  writing  narratives  for  unsuccessful  contracts  than  with  successful 
ones. 


•  What  is  the  added  value  of  the  contractor  performance  narratives 
beyond  the  value  of  the  objective  scores  for  performance? 

In  answering  the  second  research  question,  the  data  indicated  that,  overall, 
contracting  professionals  inputting  data  into  CPARS  are  doing  a  better  job  at 
providing  beneficial  data  in  the  narrative  when  the  contract  is  unsuccessful  versus 
when  it  is  successful.  Only  38.6%  of  the  sample  narratives,  whether  successful  or 
unsuccessful,  provided  a  very  good  or  exceptional  amount  of  beneficial  data  above 
and  beyond  what  could  be  gleaned  from  looking  over  the  objective  scores  assigned. 
This  shows  that  there  is  clearly  room  for  improvement  in  this  area. 

•  What  is  the  statistical  relationship  between  the  sentiment  contained  in 
the  narratives  and  the  objective  scores  for  contractor  evaluations? 

In  answering  the  third  research  question,  the  data  indicated  that,  overall, 
contracting  professionals  who  input  data  into  CPARS  are  writing  information  in  their 
narratives  that  contradicts  at  least  one  of  the  objective  scores  assigned  -8.3%  of  the 
time.  This  leaves  room  for  improvement.  Contracting  professionals  were  slightly 
better  at  matching  the  narrative  sentiment  to  the  objective  scores  in  unsuccessful 
contracts  (-81 .8%  scoring  either  very  good  or  exceptional)  than  in  successful 
contracts  (-63.2%  scoring  either  very  good  or  exceptional). 

•  To  what  degree  do  the  interview  findings  contradict,  support,  or 
enhance  the  findings  for  the  three  previous  research  questions? 

In  answering  the  fourth  research  question,  the  interviews  indicated  that  there 
are  no  issues  with  narrative  statements  to  avoid.  This  supports  the  database 
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analysis  conclusion  that  contracting  professionals  inputting  reports  into  CPARS  are 
highly  effective  at  ensuring  that  their  narratives  are  free  of  statements  to  avoid 
(-96.9%).  The  interviews  indicated  that  there  are  only  a  few  minor  issues  with 
addressing  all  performance  areas  assessed  and  ensuring  that  the  narratives  are 
based  on  objective  data.  This  supports  the  database  analysis  conclusion  that 
contracting  professionals  are  doing  a  decent  job  in  addressing  all  performance  areas 
assessed  in  their  objective  scores  (-81 .5%)  or  in  ensuring  that  the  narratives  are 
based  on  objective  data  (-77.5%).  The  interviews  indicated  that  there  are  significant 
issues  with  writing  comprehensive  narratives.  This  enhances  the  database  analysis 
conclusion  that  contracting  professionals  seem  to  have  significant  problems  with 
writing  comprehensive  narratives  (-62.5%).  The  interviews  indicated  that  there  are 
some  issues  with  writing  narratives  that  ensure  a  contracting  layman  can  understand 
the  work  performed,  especially  with  engineering-specific  entries.  This  supports  the 
database  analysis  conclusion  that  contracting  professionals  seem  to  have  significant 
problems  with  ensuring  that  a  contracting  layman  (such  as  someone  who  might 
need  to  access  CPARS  data  in  order  to  make  a  decision  about  a  future  contract 
award)  can  fully  understand  the  work  performed  (-64.5%). 

•  What  conclusions  or  recommendations  can  we  draw  from  the  answers 
to  the  previous  research  questions? 

Lastly,  in  answering  the  fifth  research  question,  we  can  conclude  that 
contracting  professionals  should  get  some  additional/remedial  training  on  writing 
robust  and  comprehensive  narratives  so  that  the  contracting  professionals  who  are 
accessing  these  reports  are  able  to  fully  understand  them.  It  may  also  be  beneficial 
for  contracting  professionals  to  get  additional  training  in  ensuring  that  the  narratives 
address  all  performance  areas  assessed  and  in  ensuring  that  the  narratives  are 
based  on  objective  data.  With  the  issues  outlined  in  this  section,  we  can  also 
recommend  that  quality  control  for  those  approving  the  narratives  posted  to  CPARS, 
as  well  as  those  using  the  data  from  PPIRS,  can  be  improved  by  sending 
unacceptable  reports  back  to  the  originator  for  correction.  We  conclude  that  there  is 
no  further  emphasis  or  training  needed  in  ensuring  that  the  narratives  are  free  of 
statements  to  avoid  since  the  data  do  not  show  that  this  is  a  significant  issue. 

2.  Additional  Findings 

In  the  process  of  completing  our  research,  especially  in  conducting  our 
interviews,  we  found  several  other  pieces  of  information  that  could  be  beneficial  to 
the  acquisition  community  with  regard  to  services  acquisition  that  we  would  like  to 
present  here. 

Finding  1:  The  information  in  CPARS/PPIRS  is  oftentimes  not  reliable,  robust, 
or  comprehensive  enough  to  allow  source  selection  officials  to  place  a  significant 
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enough  weight  on  past  performance.  This  finding  was  also  noted  in  several  GAO 
and  OFPP  memorandums  mentioned  earlier  in  the  literature  review  chapter  of  this 
report.  Without  reliable,  robust,  and  comprehensive  information,  it  is  not  possible  to 
properly  utilize  CPARS/PPIRS  in  making  fair  and  accurate  source  selection 
decisions  based  on  past  performance. 

Finding  2:  Unsuccessful  contracts  tend  to  have  more  reliable,  robust,  and 
comprehensive  past  performance  information  available  in  their  CPARS/PPIRS 
reports.  Source  selection  decisions  should  be  based  on  reliable,  robust,  and 
comprehensive  past  performance  information  for  both  successful  and  unsuccessful 
contracts  in  order  to  make  fair  and  accurate  award  decisions  based  on  past 
performance. 

Finding  3:  The  appropriate  amount  of  weight  that  should  be  assigned  to  past 
performance  in  making  a  source  selection  should  be  correlated  to  the  source, 
availability,  quality,  and  relevancy  of  the  past  performance  information.  Implementing 
this  finding  will  be  difficult  because  the  task  of  assigning  weight  to  past  performance 
information  is  normally  completed  during  the  acquisition  planning  phase  of  the 
acquisition  process.  Contractor  past  performance  is  normally  mined  later  in  the 
process  after  the  acquisition  team  has  received  proposals. 

Finding  4:  PPIRS  sometimes  contains  information  in  the  narratives  that  is 
either  contradictory  or  does  not  quite  match  up  with  the  objective  scores.  When  the 
objective  scores  and  narrative  sentiment  in  PPIRS  are  mismatched,  contracting 
professionals  tend  to  give  more  weight  to  the  narrative  versus  the  objective  scores. 
When  the  objective  scores  and  narrative  sentiment  in  PPIRS  are  mismatched,  every 
effort  should  be  made  to  send  unacceptable  reports  back  to  the  originator  for 
correction.  If  this  is  not  possible,  the  weight  assigned  to  past  performance  in  the 
source  selection  decision  should  be  reduced  to  reflect  the  fact  that  there  is  a 
contradiction  in  the  report,  thereby  reducing  its  reliability. 

Finding  5:  Contractor  past  performance  reports  might  not  be  as  accessible  as 
they  should  be  in  PPIRS  because  contracting  professionals  are  not  always  applying 
due  diligence  in  identifying  the  appropriate  entity  using  identifying  information  within 
the  contractor’s  parent  organization.  Examples  of  such  identifying  information  would 
be  an  appropriate  cage  code  or  DUNS  (Dun  and  Bradstreet  Data  Universal 
Numbering  System)  number.  It  is  imperative  that  source  selection  officials  are  able 
to  access  the  correct  reports  for  a  particular  contractor’s  past  performance  if  they 
exist.  There  must  be  a  uniform  way  of  inputting  and  accessing  the  correct 
information  in  PPIRS  if  source  selection  officials  are  to  make  fair  and  reliable  award 
decisions  using  the  past  performance  information  that  we  have  about  a  particular 
contractor. 
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Finding  6:  Source  selection  officials  often  need  to  solicit  contractors  for 
references  or  ask  them  to  fill  out  a  past  performance  questionnaire  because  there  is 
a  lack  of  reliable,  robust,  and  comprehensive  past  performance  information  available 
in  PPIRS.  The  fundamental  issue  with  using  past  performance  information  about  a 
contractor  that  is  provided  by  that  contractor  is  that  it  will  almost  certainly  only 
include  positive  references  and  omit  references  where  the  contractor  did  not  perform 
well.  When  this  is  the  only  past  performance  information  available  to  source 
selection  officials,  the  weight  assigned  to  past  performance  in  the  evaluation  criteria 
needs  to  be  reduced  to  reflect  the  reduction  in  the  reliability  and  comprehensiveness 
of  the  information. 

Finding  7:  There  has  been  a  recent  increase  in  senior-level  interest  in  the 
timely  submission  of  CPARS  reports  but  not  an  increased  interest  in  the  quality  of 
those  reports.  The  focus  needs  to  be  on  both  the  timeliness  and  quality  of  the 
reports.  Without  quality  reports,  the  information  in  the  report  will  not  be  able  to  be 
assigned  much  weight  in  future  source  selection  decisions  and  therefore  will  just  be 
a  waste  of  time  for  everyone  involved  in  the  process. 

Finding  8:  Several  of  the  interviewees  made  recommendations  for  improving 
services  acquisition  related  to  contractor  past  performance  information.  The 
recommendations  were  to  add  more  analysis  tools  and  performance  metrics  in 
PPIRS,  better  monitor  the  workload  of  CORs,  better  train  the  workforce  on  writing 
CPARS  narratives,  and  better  broadcast  the  findings  of  CPARS  Program  Office 
audits.  With  improved  analysis  tools,  appropriate  workloads  for  contracting 
professionals  inputting  the  past  performance  into  CPARS,  and  a  more  robust 
avenue  for  correcting  workforce  problem  areas,  the  past  performance  information  in 
PPIRS  will  undoubtedly  improve  and  better  assist  source  selection  officials  in 
making  fair  and  reliable  award  decisions. 

3.  Recommendations 

Based  on  our  conclusions,  we  identified  the  following  eight  recommendations. 

Recommendation  1 :  Remedial  training  should  be  implemented  for  all 
members  of  the  acquisition  teams  that  input  and  utilize  past  performance  information 
in  CPARS  and  PPIRS.  Training  should  focus  on  the  following  areas  related  to 
writing  comprehensive  narratives:  ensuring  that  acquisition  team  members  (i.e., 
someone  who  might  need  to  access  CPARS  data  in  order  to  make  a  decision  about 
a  future  contract  award)  can  fully  understand  the  work  performed,  addressing  all 
performance  areas  assessed  in  their  objective  scores,  and  ensuring  that  the 
narratives  are  based  on  objective  data. 

Recommendation  2:  There  needs  to  be  an  overall  push  for  higher  quality  past 
performance  report  submissions  in  CPARS/PPIRS  to  allow  acquisition  teams  to 
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assign  weight  to  past  performance  in  source  selection  decisions  without  having  to 
worry  about  the  source,  availability,  quality,  and  relevancy  of  the  past  performance 
information. 

Recommendation  3:  An  analysis  should  be  performed  on  the  CPARS/PPIRS 
database  by  the  program  office  to  ensure  that  contracting  professionals  are  in  fact 
able  to  accurately  input  and  retrieve  the  past  performance  information  on  the  correct 
contractor  100%  of  the  time  that  the  information  is  available. 

Recommendation  4:  A  review  of  the  CPARS/PPIRS  database  by  the  program 
office  should  be  conducted  to  see  whether  additional  analysis  tools  can  be  added  to 
better  assist  contracting  professionals  in  identifying  past  performance  trends  for  a 
particular  contractor. 

Recommendation  5:  Senior  officials  at  contracting  commands  should 
emphasize  not  only  the  timeliness  of  CPARS  submissions  but  also  emphasize  the 
quality  of  those  submissions. 

Recommendation  6:  A  requirement  should  be  added  to  the  CPARS  report 
card  approval  and  posting  process  that  allows  for  customer  feedback  on  contractor 
performance.  Agency  approving  officials  should  be  encouraged  or  required  to 
collect  (either  directly  or  via  the  COR)  and  consider  input  from  the  customer 
regarding  contractor  performance.  This  will  encourage  the  submission  of  more 
accurate  and  robust  CPARS  report  cards. 

Recommendation  7:  A  review  of  the  workload  of  CORs  should  be  conducted 
at  contracting  commands  to  ensure  they  have  the  appropriate  amount  of  time  to 
dedicate  to  the  proper  submission  of  past  performance  information. 

Recommendation  8:  The  CPARS  Program  Office  should  look  at  ways  to 
better  broadcast  the  findings  of  their  audits  to  help  the  workforce  improve  over  time. 

C.  AREAS  FOR  FURTHER  RESEARCH 

Additional  research  would  be  valuable  in  several  areas  to  further  determine 
the  value  of  the  CPAR  narratives  for  the  acquisition  process.  Obtaining  a  larger, 
more  diverse,  and  more  recent  sample  database  and  redoing  the  database  analysis 
would  add  significant  insight  into  how  the  contract  report  cards  for  other  types  of 
contracts  and  for  different  uniformed  services  are  impacting  the  acquisition  process. 
Specifically,  obtaining  a  new  database  that  contains  contracts  for  supplies  and 
contracts  for  different  military  branches,  and  comparing  CPARS  report  cards  from 
different  agencies  would  allow  for  some  interesting  comparisons  that  could  yield 
significantly  different  recommendations  for  improving  the  use  of  past  performance 
information  in  the  acquisition  process. 
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APPENDIX  A.  RESULTS  OF  THE  INTER-RATER  RELIABILITY 

TESTING 
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APPENDIX  B.  SPECIFIC  EXAMPLES  OF  NARRATIVE 
STATEMENTS  TO  AVOID  FROM  THE  CPAR  QUALITY  CHECKLIST 

(N.D.) 


Block  20:  Sample  Narrative  Statements  to  Avoid 

The  Contractor's  performance  in  this  area  was  exemplary  They  were  proactive  in  satisfying  Electrical  Kit 
Product  Performance  requirements  They  produced  a  superior  product  for  the  customer  In  many 
instances,  they  performed  engineering  tasks  outside  the  scope  of  the  contract 

•Outside  the  scope  of  the  contract  -  This  phrase  should  not  be  in  a  CPAR  narrative  It  implies  that  the 
Contractor  performed  work  not  legally  required  and  is  eligible  for  an  equitable  adjustment  to  the  contract  An 
equitable  adjustment  means  that  the  program  office/customer  will  have  to  come  up  with  additional  funds  to 
pay  for  the  additional  tasks. 

In  our  opinion,  the  Contractor's  performance  in  the  systems  engineering  area  was  very  poor  Kit  hardware 
deficiencies  were  observed  and  it  appeared  that  the  Contractor  lacked  systems  engineering  knowledge  and 
expertise  We  believe  that  some  of  the  contractual  kit  requirements  will  not  be  met  It  is  our  hope  that 
additional  factory  testing  will  eliminate  these  hardware  deficiencies  If  management  had  responded  in  a 
timely  manner,  the  requirement  might  have  been  satisfied.  Additionally  we  were  not  happy  with  the  initial 
factory  testing,  and  did  not  like  their  “fly  and  fix”  philosophy  of  testing 

*ln  our  opinion*  -  This  is  a  subjective  phrase  which  gives  the  impression  that  there  is  no  firm  evidence  to 
prove  poor  performance. 

•Appeared’  -  This  is  a  speculative  remark  which  does  not  prove  that  they  lacked  systems  engineering 
knowledge. 

*We  believe'  -  This  is  also  a  speculative  remark.  It  does  not  prove  that  they  did  not  satisfy  some  kit 
requirements 

'It  is  our  hope"  -  This  statement  does  not  belong  in  a  CPAR  narrative  The  issue  is  whether  the  Contractor 
will  correct  the  deficiencies  using  factory  testing.  If  so,  the  narrative  should  indicate  the  pending  corrections. 
If  not,  justification  should  be  provided  as  to  why  the  factory  testing  failed  to  correct  the  problems 

*We  were  not  happy”  -  This  is  an  emotional  and  subjective  statement  which  should  be  avoided  The  CPAR 
should  reflect  justification  for  the  successes/failures  from  the  factory  test 

'We  did  not  like"  -  The  customer  should  evaluate  the  results  of  the  fly  and  fix  tests  in  detail,  not  their  testing 
technique. 

The  Contractor  was  late  in  delivering  all  of  the  fOO  electric  kits  We  think  that  one  reason  is  that  their 
systems  engineering  effort  was  poor  due  to  several  electncal  component  deficiencies  Another  reason 
could  be  that  their  ability  to  manage  the  electrical  subcontracts  left  much  to  be  desired  We  established  a  6 
month  extension  to  the  contract  We  hope  they  can  deliver  the  f  00  kits  without  significant  discrepancies 

”We  think"  -  This  phrase  implies  that  the  customer  has  not  proven  the  Contractor's  poor  performance  with 
evidence. 

"Could  be’  -  This  phrase  indicates  that  the  customer  is  not  sure  that  the  reason  for  the  deficiencies  is  poor 
management.  There  is  no  proof  of  poor  management  here. 

"We  hope*  -  This  phrase  implies  that  the  delivery  of  the  kits  without  deficiencies  in  the  time  penod  allotted  is 
a  desire,  not  a  contractual  requirement _ 
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APPENDIX  C.  AN  EXAMPLE  OF  AN  INSUFFICIENT  NARRATIVE 
VS.  A  BETTER  WAY  TO  WRITE  THIS  TAKEN  FROM  THE  CPAR 

QUALITY  CHECKLIST  (N.D.) 


Block  20:  Sample  CPAR  Narrative 


Insufficient  Narrative: 

Schedule:  Very  Good  -  In  our  opinion,  the  Contractor  has  done  really  well  in  terms  of 
schedule  The  Training  Manager  Jack  Jones,  is  pleasant  and  easy  to  work  with.  He 
adapts  to  our  schedule  changes  amazingly  and  never  complains.  He  also  went  above 
and  beyond  and  fixed  our  pnnter  and  fax  without  charging  the  Government  and  he 
continued  to  meet  all  the  contract  objectives  in  the  interim  Great  job! 

The  example  above  is  missing  detail  to  support  the  score  and  supporting  documentation 
and  metrics.  In  addition,  it  uses  an  individual's  name,  addresses  worfc  outside  the 
contract  scope,  and  uses  subjective  phrases 

A  Better  Way  to  Write  This: 

Schedule:  Very  Good  -  Contractor  successfully  executed  the  delivery  and  training 
requirements  for  this  penod  ahead  of  schedule.  For  example,  there  were  20  training  site 
visits  scheduled  for  this  period  however,  the  Contractor  conducted  31  visits  in  the  same 
period  of  time  The  Contractor  also  met  100%  of  the  13  contract  data  requirements  in  a 
45  day  timeframe  versus  the  60  days  allotted.  This  resulted  in  data  requirements  14-20 
being  completed  earlier  than  anticipated  This  was  done  with  minimal  supervision  by  the 
program  office  hence  allowing  more  time  for  additional  projects.  A  20  site  preventative 
maintenance  visit  ran  behind  schedule  for  the  first  8  months  of  the  reporting  penod  due  to 
equipment  failures,  but  Contractor  management  was  able  to  bnng  the  visit  back  on 
schedule  due  to  implementation  of  an  aggressive  quality  management  system  and 
spares  availability  policy. _ 
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APPENDIX  D.  INTERVIEW  QUESTIONS  FOR  DOD 
CONTRACTING  PROFESSIONALS 


INTERVIEW  INTRODUCTION 

The  following  section  will  be  read  verbatim  to  each  interview  subject  in  order 
to  explain  to  them  the  purpose  of  the  interview  and  to  ask  their  permission  to 
perform  the  interview. 

Project  Title: 

Determining  the  Value  of  Performance  Evaluation  Narratives  for  Service 
Contracts 

Purpose  of  Interview: 

We  are  three  graduate  students  enrolled  in  the  Acquisition  and  Contract 
Management  Curriculum  at  the  Naval  Postgraduate  School.  After  graduating,  we 
will  be  working  in  the  contracting  field  for  the  Department  of  Defense.  We  are 
conducting  research  in  an  effort  to  determine  the  value  of  contractor  performance 
assessment  report  narratives  for  services  contracts.  In  the  Department  of  Defense, 
members  of  the  acquisition  team  submit  contractor  past  performance  information  to 
the  Contractor  Performance  Assessment  Reporting  System  (CPARS).  This 
information  includes  objective  (numeric)  scores  for  contractor  performance  in  five 
categories  as  well  as  a  written  narrative.  Contracting  professionals  and  source 
selection  team  members  are  able  to  retrieve  contractor  past  performance 
information  that  was  submitted  to  CPARS  through  the  Past  Performance  Information 
Retrieval  System  (PPIRS). 

We  have  studied  and  conducted  statistical  analysis  on  CPARS  contractor 
past  performance  narratives  and  how  they  relate  to  their  associated  objective 
scores.  As  part  of  this  research,  we  would  like  to  ask  for  your  assistance  by  allowing 
us  to  interview  you.  The  purpose  of  the  interview  is  to  obtain  data  about  how  and  to 
what  extent  you  use  the  objective  scores  and  narratives  when  reviewing  CPARS  (or 
PPIRS)  report  cards  for  source  selection  and  contract  administration.  This  interview 
data,  combined  with  a  statistical  analysis  of  narratives  and  objectives  scores,  should 
allow  us  to  make  recommendations  for  improving  services  acquisition  and 
streamlining  the  way  contracting  professionals  submit  to  and  utilize  these  systems. 
Our  goal  is  to  make  these  programs  more  useful  to  you,  the  contracting  professional, 
as  well  as  easier  to  use,  if  there  is  any  improvement  to  be  had. 

Your  command  and  department  head  have  invited  us  here,  upon  our  request, 
to  conduct  this  interview  and  you  have  been  selected  by  them  to  be  a  potential 
interviewee  based  on  your  role  and  experience  as  a  contracting  professional. 
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Participation  in  the  interview  is  voluntary  and  anonymous.  Your  responses  to  our 
questions  will  not  be  able  to  be  traced  back  to  you  or  your  command  and  they  will 
not  be  reported  to  your  supervisors  or  chain-of-command.  This  interview  consists  of 
13  questions  and  should  take  about  30  minutes  to  complete. 

Do  you  agree  to  assist  us  in  conducting  this  research  by  allowing  us  to 
interview  you? 

Goal  of  the  Interview: 

The  previous  section  Purpose  of  Interview  was  formulated  with  the  following 
goals  in  mind: 

•  Brevity.  We  do  not  want  the  subjects  to  lose  interest  before  the 
interview  begins. 

•  Clarity.  This  introduction  to  our  research  is  concise  and  detailed 
enough  to  be  readily  understood  by  a  contracting  professional  with 
enough  knowledge  of  the  subject  matter  to  be  useful  in  our  research. 

•  Unbiased.  We  do  not  want  to  show  whether  we  believe  contractor  past 
performance  objective  scores  have  any  more  or  less  value  than  the 
associated  narratives.  We  do  not  want  to  specify  whether  there  is  in 
fact  any  way  to  improve  upon  the  CPARS/PPIRS  programs.  In  an 
effort  to  avoid  encouraging  a  self-serving  bias  in  interview  subjects  we 
do  not  want  to  specify  that  submitting  contractor  past  performance 
information  to  CPARS  is  directed  by  the  Department  of  Defense.  We 
want  to  get  them  to  answer  questions  honestly  without  concern  that 
they  might  not  have  been  closely  following  protocol. 

•  Encouraging.  We  want  to  identify  with  our  subjects  as  fellow 
government  contracting  professionals  with  a  mutual  interest  in 
improving  our  processes  and  systems.  We  want  to  encourage  them  to 
agree  to  the  interview  and  to  be  forthcoming  because  in  doing  so  they 
might  help  all  of  us  in  the  future. 

•  Non-retribution.  Subjects  should  be  clear  on  the  fact  that  there  will  be 
no  way  that  this  interview  can  come  back  to  hurt  them  and  that  their 
identities  will  be  completely  protected.  Interviews  are  completely 
voluntary  and  encouraged  by  the  interviewee’s  command. 

INTERVIEW  QUESTIONS 

How  often  do  you  use  contractor  past  performance  information  for 
source  selections? 

[Provide  Verbal  Categories  of  Options  for  Answers] 
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This  question  allows  us  to  approximate  the  level  of  depth  of  familiarity  the 
subject  has  with  contractor  past  performance  information  in  general  and  specifically 
whether  he  or  she  participates  in  contract  source  selections.  This  question  is 
carefully  worded  in  order  to  avoid  asking  alternative  questions  such  as,  How  often 
do  you  access  PPIRS?  What  portion  of  the  time,  when  conducting  source  selection, 
do  you  use  contractor  past  performance  information?  We  want  to  know  specifically 
how  many  times  in  a  given  period  they  use  contractor  past  performance  information 
for  source  selection  in  order  to  determine  the  subject’s  depth  of  experience  and  to 
determine  to  what  extent  contractor  past  performance  information  is  used  as 
compared  to  how  frequently  it  is  used  during  contract  administration. 

In  order  to  help  the  subject  understand  the  question  we  will  provide  examples 
of  potential  answers,  carefully  in  order  to  avoid  injecting  bias,  such  as,  once  a  year, 
or  several  times  a  day.  Using  examples  on  the  extreme  ends  of  the  range  will  keep 
us  from  planting  a  likely  answer  in  the  mind  of  the  subject. 

How  often  do  you  use  contractor  past  performance  information  for 
contract  administration? 

[Provide  Verbal  Categories  of  Options  for  Answers] 

This  question  is  the  same  as  the  previous  question  but  for  contract 
administration  instead  of  source  selection.  The  purposes  of  this  question  are  the 
same  as  with  the  previous  question  except  for  isolating  contract  administration  from 
source  selection.  We  do  not  believe  it  would  be  sufficient  to  ask  one  question:  How 
often  do  you  use  contractor  past  performance  information  for  contract  administration 
and/or  source  selection?  For  the  purposes  of  the  research  we  want  to  know  to  what 
extent  it  is  used  for  each  specific  purpose. 

When  inputting  past  performance  data  into  CPARS,  what  is  the  typical 
amount  of  time  spent  per  contract  gathering  information  and  inputting  it  into 
the  system? 

There  is  some  purposeful  ambiguity  in  this  question.  Some  contracting 
professionals  write  the  entire  narrative  for  one  CPARS  report  card  while  in  other 
cases,  especially  for  large  and  complex  acquisitions,  the  task  is  shared  by  up  to 
several  members  of  the  acquisition  team.  We  want  to  allow  the  interview  subject  to 
specify  how  much  time  is  spent  per  contract  gathering  and  inputting  information 
whether  or  not  all  of  the  effort  was  conducted  by  the  interviewee.  This  may  require 
that  the  subject  make  some  estimates  of  how  much  time  other  individuals  are 
spending  on  this  task. 

Does  this  amount  of  time  vary  greatly  or  is  it  usually  around  the  typical 
amount  of  time? 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


■67- 


The  purpose  of  this  question  is  to  establish  whether  there  is  a  lot  of  variability 
in  the  amount  of  time  one  spends  putting  together  a  CPARS  submission.  If  there  is 
not  much  variability  then  the  follow-up  question  would  be  to  determine  whether  there 
is  much  variability  in  the  complexity  of  the  contracts  the  subject  handles.  If  there  is  a 
lot  of  variability,  we  want  to  determine  whether  the  variability  is  based  primarily  on 
the  size  and  complexity  of  the  contracts.  We  clarify  the  impact  of  the  data  from  the 
previous  question  with  one  of  the  two  following  questions. 

[ask  the  following  question  if  subject  responded  with  answer  more 
similar  to  “the  amount  of  time  is  usually  around  the  typical  amount  of  time”] 

Are  the  contracts  you  write  or  administer  usually  relatively  similar  in 
size  and  complexity  or  do  they  tend  to  vary  greatly? 

[ask  the  following  question  if  subject  responded  with  answer  more 
similar  to  “the  amount  of  time  varies  greatly”] 

Do  you  typically  find  that  gathering  and  submitting  information  to 
CPARS  takes  more  time  for  larger  more  complex  contracts  or  do  you  find  that 
the  amount  of  time  varies  not  related  to  the  size  and  complexity  of  the 
contract? 

Subjects  will  answer  one  of  the  two  previous  questions. 

Typically,  what  portion  of  the  total  time  it  takes  to  prepare  and  submit  a 
CPARS  report  is  spent  preparing  and  writing  the  narrative  portion  of  it? 

The  purpose  of  this  question  is  to  determine  what  portion  of  their  time  is  spent 
preparing  the  narrative  which  will  be  one  of  the  evaluation  factors  considered  when 
determining  the  relative  value  of  the  narrative.  Our  evaluation  will  compare  the 
relative  value  of  the  narratives  to  the  relative  amount  of  effort  it  takes  to  create  them. 

What  is  the  typical  number  of  people  involved  per  contract  in  entering 
contractor  past  performance  information  into  CPARS? 

The  purpose  of  this  question  is  to  tell  us  the  typical  size  of  the  acquisition 
team.  This  provides  us  data  useful  in  analyzing  the  CPARS  written  narratives. 

Some  narratives  are  very  uniform  throughout  their  sections  in  terms  of  positive  or 
negative  sentiment,  while  others  have  much  more  variation.  Some  CPARS  written 
narratives  are  written  by  one  individual  while  others  are  written  in  parts  by  several. 
Useful  data  will  show  whether  it  is  much  more  or  much  less  likely  that  an  individual 
written  narrative  was  written  by  one  person  than  more  than  one  person. 

[Ask  the  following  question  if  the  answer  to  the  previous  question  was 
greater  than  one] 
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Which  members  of  the  acquisition  team  (contracting  officer,  COR, 
program  manager,  requirements  manager,  etc.)  are  responsible  for  entering 
past  performance  information  into  CPARS? 

The  purpose  of  this  question  is  to  determine  specifically  which  members  of 
the  acquisition  team  are  most  likely  to  participate  in  the  writing  of  a  CPARS 
narrative. 

When  you  are  conducting  a  source  selection  or  administering  a  contract 
does  contractor  past  performance  information  have  any  impact? 

The  purpose  of  this  question  is  to  determine  to  what  extent,  in  general,  the 
interview  subject  values  contractor  past  performance  information.  In  this  case,  we 
feel  that  combining  source  selection  and  contract  administration  will  provide 
adequately  specific  and  useful  data  for  our  research  purposes.  The  question  is 
specifically  worded  to  avoid  injecting  bias  and  will  apply  to  all  interview  subjects 
irrespective  of  answers  to  previous  interview  questions.  If  the  interview  subject 
answers  in  the  negative  then  there  is  no  need  for  a  follow-on  question. 

[If  answer  to  previous  question  was  in  the  affirmative,  ask  the  following 
question] 

To  what  extent  does  contractor  past  performance  information  impact 
your  source  selection  or  contract  administration? 

The  purpose  of  this  question  is  to  determine  the  value  of  CPARS/PPIRS  to 
the  individual  contracting  professional.  Contracting  officers  and  source  selection 
teams  are  allowed  by  the  Federal  Acquisition  Regulation  (FAR)  to  use  some 
subjectivity  in  determining  how  much  to  consider  contractor  past  performance. 
Knowing  how  much  they  value  the  product  will  aid  us  in  determining  whether 
improvements  to  the  system  might  be  beneficial  and  whether  the  quality  of  the 
system  affects  how  much  value  is  assigned  to  it. 

Which  provides  more  impact,  the  objective  scores  or  the  narratives? 

The  purpose  of  this  question  is  to  compare  the  two  sections  of  the  CPARS 
Report  Card  in  terms  of  value  to  the  contracting  professional.  This  question  is 
carefully  worded  so  that  it  is  a  zero  sum  value  pool  split  between  the  two  sections. 
We  are  not  asking  how  much  total  value  each  section  provides  independent  of  each 
other. 

What  ideas  do  you  think  would  increase  the  impact  of  contractor  past 
performance  information  in  source  selection  and/or  contract  administration? 

The  purpose  of  this  question  is  to  get  opinions  from  the  interview  subject. 
Since  evaluating  past  performance  information  is  mandated  by  the  FAR,  our 
research  is  attempting  to  find  ways  in  which  CPARS/PPIRS  can  be  made  more  user- 
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friendly  and  responsive  for  this  purpose.  Opinions  from  experts  in  the  field  will  be 
very  useful. 

Do  you  feel  that  past  performance  information  is  weighted  appropriately 
in  source  selection? 

The  purpose  of  this  question  is  to  determine  whether  the  interview  subject 
believes  that  the  FAR  or  their  agency  puts  too  much  or  too  little  emphasis  on 
contractor  past  performance.  If  the  interview  subject  believes  there  is  too  much 
emphasis  on  past  performance,  it  might  be  because  the  CPARS/PPIRS  systems  are 
insufficient  for  providing  useful  information.  This  question  was  worded  to  avoid  the 
injection  of  bias.  The  data  obtained  from  the  answers  to  this  question  will  be 
particularly  useful  when  combined  with  the  information  from  the  other  answers 
received. 

Do  you  find  that  the  narrative  usually  provides  useful  information  above 
and  beyond  the  objective  scores? 

The  purpose  of  this  question  is  to  determine  whether  the  CPARS  narrative 
section  has  value  and  whether  or  not  the  objective  scores  have  more  value  than  the 
narrative.  It  is  possible  that  the  interview  subject  will  answer  yes  even  if  he  or  she 
answered  to  a  previous  question  that  the  objective  scores  are  more  impactful  than 
the  narrative. 

If  the  objective  scores  of  past  performance  information  were  to  be 
eliminated,  how  would  that  impact  the  source  selection  or  contract 
administration  process? 

The  purpose  of  this  question  is  to  establish  the  absolute  value  of  the  CPARS 
objective  scores.  It  allows  the  interview  subject  the  opportunity  to  give  a  bottom-line 
answer  to  the  question,  How  much  does  this  really  matter?  The  answers  to  this 
question  are  going  to  be  particularly  useful  when  compared  to  answers  to  previous 
interview  questions.  If  the  interviewee  finds  that  the  narrative  section  is  much  more 
impactful  than  the  objective  scores  then  the  conclusion  will  be  far  different  than  if  the 
opposite  is  true. 

If  the  narrative  portion  of  past  performance  information  were  to  be 
eliminated,  how  would  that  impact  the  source  selection  or  contract 
administration  process? 

The  purpose  of  this  question  is  the  same  as  the  previous  question  except  for 
the  narrative  portion  of  the  CPARS  Report  Card. 
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